2

I have recently been using the Mechanize gem in ruby to write a scraper. Unfortunately, the URL that I am attempting to scrape returns a Mechanize::File object instead of a Mechanize::Page object upon a GET request.

I can't figure out why. Every other URL I have tried has returned a Mechanize::Page object.

Is there some way to force Mechanize to return a Page object?

2
  • 1
    Give the two examples of pages which give the different results
    – kwicher
    Aug 3, 2011 at 0:40
  • 1
    This could be something the target server is doing based upon the User-Agent or something like that.
    – mjc
    Aug 3, 2011 at 3:37

3 Answers 3

6

Here's what's going on.

When you download a 'normal' web page, its header will have a field that says something like Content-Type text/html. When Mechanize sees this, it knows to interpret the page content as HTML and parses it into a Mechanize::Page object, complete with links and forms and whatnot.

But if you've ever clicked on a link that says "download CSV data" or "download PDF" or, in short, anything that's not HTML, you're receiving a page that does not have a Content-Type of text/html. Since Mechanize cannot parse non-html into a Mechanize::Page object, it will package up the content into a Mechanize::File object.

What you do with the Mechanize::File object depends on what you're trying to accomplish. For example, if you know that the page you visited was CSV data rather than HTML, you can extract the CSV data like this:

page = web_agent.get(some_url_that_references_csv_data)
parsed_csv = CSV.parse(page.body)

If you want to be fancy, you can write your own parsers that allow Mechanize to handle non-HTML formats. See the Mechanize docs on PluggableParser if you want to go that route. But you can accomplish plenty by working directly with the Mechanize::File object.

addendum in response to @user741072's comment

If on the other hand, if the page is HTML and somebody neglected to set content-type to HTML, you can write a method that swaps in the html parser for the default parser just long enough to parse the page. This will force parsing as HTML:

def with_html_parser(agent, &body)
  original_parser = agent.pluggable_parser.default
  agent.pluggable_parser.default = agent.pluggable_parser['text/html']
  begin
    yield
  ensure
    agent.pluggable_parser.default = original_parser
  end
end

Let me know if that does the trick.

2
  • To be totally fair, my answer isn't really saying anything that @zhon and gogogopenny haven't already said -- I'm just stating it in simpler terms. Apr 10, 2012 at 19:45
  • It's a html page. The problem is the moron who wrote it didnt state that in the header. Can I force Mechanize to interpret it as a HTML file?
    – JRPete
    Apr 13, 2012 at 7:18
3

When a website doesn't returns the Content Type as part of the response, you can set the Content Type yourself in a post connection hook:

agent = Mechanize.new { |a|
  a.post_connect_hooks << lambda { |_,_,response,_|
    if response.content_type.nil? || response.content_type.empty?
      response.content_type = 'text/html'
    end
  }
}
4
  • That's exactly what I needed. Small amendment:response.content_type may be nil, so you need to test for that, e.g. using blank?. Feb 15, 2012 at 0:15
  • 1
    Thanks. BTW "".blank? is part of rails not ruby.
    – zhon
    Mar 19, 2012 at 20:08
  • What? You can run ruby without rails?? ;) Apr 6, 2012 at 16:48
  • Yeah, weird how a library is more popular than the language it is written in.
    – zhon
    Apr 6, 2012 at 19:13
-1

have a look out for the Content-Type of the specific url in the HTTP-Headers with curl (curl yoururl -i). In your code you might want to check the content-type before you get the url:

require 'net/http'
url = URI.parse('http://www.gesetze-im-internet.de/bundesrecht/bgb/gesamt.pdf')
res = Net::HTTP.start(url.host, url.port) {|http| http.head(url.path)}
puts res['content-type']

#=> application/pdf

Or you can just check if your Mechanize object is of class Mechanize::Page:

agent = Mechanize.new
unknown_body = agent.get(url)
if unknown_body.class == Mechanize::Page
  self.body = unknown_body
else
  puts "Discarded binary content!"
end

Be aware that this approach will be much slower since it "downloads" the requested resource anyways.But it might be useful if you want to store the file for later usage.

2
  • Mechanize supports head requests. Mar 19, 2012 at 20:21
  • You can retrieve the Content-Type from Mechanize::File and you can install pluggable parsers as described above. There's no need to do the extra work, mechanize can do it all for you
    – drbrain
    May 7, 2012 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.