55

Is there a way to get around the following?

httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt

Is the only way around this to contact the site-owner (barnesandnoble.com).. i'm building a site that would bring them more sales, not sure why they would deny access at a certain depth.

I'm using mechanize and BeautifulSoup on Python2.6.

hoping for a work-around

1
  • There are probably legal issues if you plan to monetize, but if you don't, continue as you please. Long live scroogle. Commented May 17, 2010 at 0:44

8 Answers 8

240

oh you need to ignore the robots.txt

br = mechanize.Browser()
br.set_handle_robots(False)
2
  • 1
    That's exactly what I was looking for. Commented May 6, 2016 at 19:48
  • wow this still works! I wonder if this is illegal in any way. Commented Nov 24, 2019 at 23:13
15

You can try lying about your user agent (e.g., by trying to make believe you're a human being and not a robot) if you want to get in possible legal trouble with Barnes & Noble. Why not instead get in touch with their business development department and convince them to authorize you specifically? They're no doubt just trying to avoid getting their site scraped by some classes of robots such as price comparison engines, and if you can convince them that you're not one, sign a contract, etc, they may well be willing to make an exception for you.

A "technical" workaround that just breaks their policies as encoded in robots.txt is a high-legal-risk approach that I would never recommend. BTW, how does their robots.txt read?

3
  • Their robots.txt only disallows "/reviews/reviews.asp" - is this what you are scraping?
    – fmark
    Commented May 17, 2010 at 2:43
  • Thanks Alex, I agree... after reading more about robots.txt, this is the best approach. Cheers... @fmark i'm scraping off the video portion... video.barnesandnoble.com/robots.txt
    – Diego
    Commented May 18, 2010 at 0:38
  • 15
    robots.txt is not legally binding. (nytimes.com/2005/07/13/technology/…)
    – markwatson
    Commented May 2, 2011 at 0:54
13

The code to make a correct request:

br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
resp = br.open(url)
print resp.info()  # headers
print resp.read()  # content
1
  • 1
    The only answer that explains - how do we set headers along with disabling robots.txt handling.
    – markroxor
    Commented Jan 3, 2018 at 6:20
5

Mechanize automatically follows robots.txt, but it can be disabled assuming you have permission, or you have thought the ethics through ..

Set a flag in your browser:

browser.set_handle_equiv(False) 

This ignores robots.txt.

Also, make sure you throttle your requests, so you don't put too much load on their site. (Note, this also makes it less likely that they will detect and ban you).

3
  • Hey wisty, what do you mean by throttle your requests?
    – Diego
    Commented May 18, 2010 at 0:39
  • I mean, set a small timeout after each request (i.e. time.sleep(1)), and don't use many threads. I'd use a few threads (in case some get bogged down), and a few seconds sleep.
    – wisty
    Commented May 18, 2010 at 1:21
  • 1
    this didn't work with the current version of mechanize Commented Oct 24, 2014 at 13:31
3

The error you're receiving is not related to the user agent. mechanize by default checks robots.txt directives automatically when you use it to navigate to a site. Use the .set_handle_robots(false) method of mechanize.browser to disable this behavior.

1

Set your User-Agent header to match some real IE/FF User-Agent.

Here's my IE8 useragent string:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; AskTB5.6)
0
0

Without debating the ethics of this you could modify the headers to look like the googlebot for example, or is the googlebot blocked as well?

2
  • I don't see any ethical problem but the legal ones could get even worse (whoever you're impersonating could detect you and sue the expletive-deleted out of you, not just B&N and your ISP). "Do this illegal thing and just don't get caught" isn't prudent advice, even when no ethical issues pertain (and, I repeat, I don't see anything immoral in breaking these particular laws -- it's just too risky for far too little potential gain;-). Commented May 17, 2010 at 0:51
  • A legal issue is an ethical issue in this case do you follow it or not. Commented May 17, 2010 at 0:53
0

As it seems, you have to do less work to bypass robots.txt, at least says this article. So you might have to remove some code to ignore the filter.

1
  • That article is more about custom code to scrape websites. If you are using some library, the library might be already respecting robots.txt.
    – Niyaz
    Commented Oct 16, 2012 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.