55

Is there a way to get around the following?

httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt

Is the only way around this to contact the site-owner (barnesandnoble.com).. i'm building a site that would bring them more sales, not sure why they would deny access at a certain depth.

I'm using mechanize and BeautifulSoup on Python2.6.

hoping for a work-around

1
  • There are probably legal issues if you plan to monetize, but if you don't, continue as you please. Long live scroogle. May 17, 2010 at 0:44

8 Answers 8

241

oh you need to ignore the robots.txt

br = mechanize.Browser()
br.set_handle_robots(False)
2
  • 1
    That's exactly what I was looking for. May 6, 2016 at 19:48
  • wow this still works! I wonder if this is illegal in any way. Nov 24, 2019 at 23:13
16

You can try lying about your user agent (e.g., by trying to make believe you're a human being and not a robot) if you want to get in possible legal trouble with Barnes & Noble. Why not instead get in touch with their business development department and convince them to authorize you specifically? They're no doubt just trying to avoid getting their site scraped by some classes of robots such as price comparison engines, and if you can convince them that you're not one, sign a contract, etc, they may well be willing to make an exception for you.

A "technical" workaround that just breaks their policies as encoded in robots.txt is a high-legal-risk approach that I would never recommend. BTW, how does their robots.txt read?

4
  • Their robots.txt only disallows "/reviews/reviews.asp" - is this what you are scraping?
    – fmark
    May 17, 2010 at 2:43
  • Thanks Alex, I agree... after reading more about robots.txt, this is the best approach. Cheers... @fmark i'm scraping off the video portion... video.barnesandnoble.com/robots.txt
    – Diego
    May 18, 2010 at 0:38
  • 14
    robots.txt is not legally binding. (nytimes.com/2005/07/13/technology/…)
    – markwatson
    May 2, 2011 at 0:54
  • In the US that may be right (the result of the law suit isn't given and the people giving their opinions may not be a representative sample anyway), but laws vary considerably across the world. In the UK it may well be a criminal offence to do what is being asked since it may well be contrary to s.1 of the Computer Misuse Act 1990. This may not be a problem for Diego, but I would counsel caution. Jan 27, 2014 at 20:07
13

The code to make a correct request:

br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
resp = br.open(url)
print resp.info()  # headers
print resp.read()  # content
1
  • 1
    The only answer that explains - how do we set headers along with disabling robots.txt handling.
    – markroxor
    Jan 3, 2018 at 6:20
5

Mechanize automatically follows robots.txt, but it can be disabled assuming you have permission, or you have thought the ethics through ..

Set a flag in your browser:

browser.set_handle_equiv(False) 

This ignores robots.txt.

Also, make sure you throttle your requests, so you don't put too much load on their site. (Note, this also makes it less likely that they will detect and ban you).

3
  • Hey wisty, what do you mean by throttle your requests?
    – Diego
    May 18, 2010 at 0:39
  • I mean, set a small timeout after each request (i.e. time.sleep(1)), and don't use many threads. I'd use a few threads (in case some get bogged down), and a few seconds sleep.
    – wisty
    May 18, 2010 at 1:21
  • 1
    this didn't work with the current version of mechanize Oct 24, 2014 at 13:31
3

The error you're receiving is not related to the user agent. mechanize by default checks robots.txt directives automatically when you use it to navigate to a site. Use the .set_handle_robots(false) method of mechanize.browser to disable this behavior.

1

Set your User-Agent header to match some real IE/FF User-Agent.

Here's my IE8 useragent string:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; AskTB5.6)
0
0

Without debating the ethics of this you could modify the headers to look like the googlebot for example, or is the googlebot blocked as well?

2
  • I don't see any ethical problem but the legal ones could get even worse (whoever you're impersonating could detect you and sue the expletive-deleted out of you, not just B&N and your ISP). "Do this illegal thing and just don't get caught" isn't prudent advice, even when no ethical issues pertain (and, I repeat, I don't see anything immoral in breaking these particular laws -- it's just too risky for far too little potential gain;-). May 17, 2010 at 0:51
  • A legal issue is an ethical issue in this case do you follow it or not. May 17, 2010 at 0:53
0

As it seems, you have to do less work to bypass robots.txt, at least says this article. So you might have to remove some code to ignore the filter.

1
  • That article is more about custom code to scrape websites. If you are using some library, the library might be already respecting robots.txt.
    – Niyaz
    Oct 16, 2012 at 4:39

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.