0

I am attempting to write a web scraper in Ruby/Mechanize. One thing I am trying to implement is a function which can find potential URL Parameters. Here's a snippet:

require 'mechanize'
def find_parameters(url)
    mechanize = Mechanize.new
    result = []
    # build list of potential parameters at URL
    result # return
end

Imagine sending passing in the URL http://example.com/. On example.com there is an index.php file which takes in a URL parameter call baz, and prints the value of that parameter to the page.

<?php
    if (isset($_GET['baz'])) {
        echo $_GET['baz'];
    }
?>

Thus http://example.com?baz=123 will go to a page that prints 123. We know looking through the source code that baz is a potential paramter, is there any way to get Mechanize to find all the potential parameters and return a list of them?

Ex: find_parameters('http://example.com/') => ['baz']

3
  • 1
    Note: Filenames in require are almost always lower-case. This may work on a case-insensitive filesystem, but will break on a case-sensitive one.
    – tadman
    Sep 19, 2019 at 19:43
  • 1
    ^ noted and fixed
    – erli
    Sep 19, 2019 at 19:49
  • Nope, impossible. If the page is not documenting its parameters in any way, you can't find out what they are. Sep 19, 2019 at 21:26

1 Answer 1

0

You can tweak the string:

require 'mechanize'
def find_parameters(url)
  mechanize = Mechanize.new
  result = []
  mechanize.get(url)  #go to the page
  # get the current page, split in the possible parameters list, split by parameters
  # (rescue in case there are no params)
  ( mechanize.page.uri.to_s.split("?")[1].split("&") rescue []).each do |key_val| 
    # split the pair of param and value, and store the param name
    result << key_val.split("=")[0]
  end
  return result
end
3
  • Huh? Here the parameters are already in the url. No need to load the page, just parse the url string. Sep 20, 2019 at 11:20
  • it's possible that www.example.com, when loaded returns the www.example.com?baz=123, at least this is how I interpret it.
    – Mejs
    Sep 20, 2019 at 14:12
  • The provided php example will not do that. Question is how can one find those "hidden" parameters? Sep 20, 2019 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.