14

As per these instructions I can see that HTTP 500 errors, connection lost errors etc. are always rescheduled but I couldn't find anywhere if 403 error are rescheduled too or if they are simply treated as a valid response or ignored after reaching the retry limits.

Also from the same instruction:

Failed pages are collected on the scraping process and rescheduled at the end, once the spider has finished crawling all regular (non failed) pages. Once there are no more failed pages to retry, this middleware sends a signal (retry_complete), so other extensions could connect to that signal.

What does these Failed Pages refer to ? Do they include 403 errors ?

Also, I can see this exception being raised when scrapy encounters a HTTP 400 status:

2015-12-07 12:33:42 [scrapy] DEBUG: Ignoring response <400 http://example.com/q?x=12>: HTTP status code is not handled or not allowed

From this exception I think it's clear that HTTP 400 responses are ignored and not rescheduled.

I'm not sure if 403 HTTP status is ignored or rescheduled to be crawled at the end. So I tried rescheduling all the responses that have HTTP status 403 according to these docs. Here's what I have tried so far:

In a middlewares.py file:

def process_response(self, request, response, spider):
    if response.status == 403:
        return request
    else:
        return response

In the settings.py:

RETRY_TIMES = 5
RETRY_HTTP_CODES = [500, 502, 503, 504, 400, 403, 404, 408]

My questions are:

  1. What does these Failed Pages refer to ? Do they include 403 errors ?
  2. Do I need to write process_response to reschedule 403 error pages or are they automatically rescheduled by scrapy ?
  3. What type of exceptions and (HTTP codes) are rescheduled by scrapy ?
  4. If I reschedule a 404 error page, will I be entering an infinite loop or is there a timeout after which the rescheduling will not be done further ?

2 Answers 2

14
  1. You can find the default statuses to retry here.

  2. Adding 403 to RETRY_HTTP_CODES in the settings.py file should handle that request and retry.

  3. The ones inside the RETRY_HTTP_CODES, we already checked the default ones.

  4. The RETRY_TIMES handles how many times to try an error page, by default it is set to 2, and you can override it on the settings.py file.
7
  • Hey, Thanks for the response. I pretty much know about retry. My main question is about rescheduling. Are the codes that are retried are also rescheduled ?
    – Rahul
    Commented Dec 8, 2015 at 4:16
  • I don't really understand what you mean, but retrying is launching the Request again.
    – eLRuLL
    Commented Dec 8, 2015 at 4:19
  • 1
    Failed pages are collected on the scraping process and rescheduled at the end, once the spider has finished crawling all regular (non failed) pages. I'm talking about this rescheduling.
    – Rahul
    Commented Dec 8, 2015 at 4:21
  • 1
    Cool. So what happens when the retry limit is over ? Does it discard the request by raising an exception or it keeps on retrying until it gets a valid response ?
    – Rahul
    Commented Dec 8, 2015 at 4:33
  • 1
    It goes to the errback method specified on the Request object
    – eLRuLL
    Commented Dec 8, 2015 at 4:38
0

One way would be to add a middleware to your Spider (source, linked):

# File: middlewares.py

from twisted.internet import reactor
from twisted.internet.defer import Deferred

DEFAULT_DELAY = 5

class DelayedRequestsMiddleware(object):
    def process_request(self, request, spider):
        delay_s = request.meta.get('delay_request_by', None)
        if not delay_s and response.status != 403:
            return
        delay_s = delay_s or DEFAULT_DELAY

        deferred = Deferred()
        reactor.callLater(delay_s, deferred.callback, None)
        return deferred

In the example you could invoke the delay with meta key:

# This request will have itself delayed by 5 seconds
yield scrapy.Request(url='http://quotes.toscrape.com/page/1/', 
                     meta={'delay_request_by': 5})

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.