I'm trying to scrape a website for broken links, so far I have this code which is successfully logging in and crawling the site, but it's only recording HTTP status 200 codes:
class HttpStatusSpider(scrapy.Spider):
name = 'httpstatus'
handle_httpstatus_all = True
link_extractor = LinkExtractor()
def start_requests(self):
"""This method ensures we login before we begin spidering"""
# Little bit of magic to handle the CSRF protection on the login form
resp = requests.get('http://localhost:8000/login/')
tree = html.fromstring(resp.content)
csrf_token = tree.cssselect('input[name=csrfmiddlewaretoken]')[0].value
return [FormRequest('http://localhost:8000/login/', callback=self.parse,
formdata={'username': 'mischa_cs',
'password': 'letmein',
'csrfmiddlewaretoken': csrf_token},
cookies={'csrftoken': resp.cookies['csrftoken']})]
def parse(self, response):
item = HttpResponseItem()
item['url'] = response.url
item['status'] = response.status
item['referer'] = response.request.headers.get('Referer', '')
yield item
for link in self.link_extractor.extract_links(response):
r = Request(link.url, self.parse)
r.meta.update(link_text=link.text)
yield r
The docs and these answers lead me to believe that handle_httpstatus_all = True
should cause scrapy to pass errored requests to my parse
method, but so far I've not been able to capture any.
I've also experimented with handle_httpstatus_list
and a custom errback
handler in a different iteration of the code.
What do I need to change to capture the HTTP error codes scrapy is encountering?
allowed_domains
argument, it isn't needed and it could also filter your requests, maybe that's the problemallowed_domains = ['localhost']
with no change in behaviourallowed_domains = ['localhost']
back in, after the spider ended up finding its way onto tripadvisor:2018-12-17 19:29:09 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.tripadvisor.co.uk/Restaurants-g186364-c31-zfp5-Sheffield_South_Yorkshire_England.html>