3

I'm trying to make requests retry if the response status code is not 200, so I wrote this function:

    def _get_resource(url, max_retries=0, delay=0):
         number_retries = 0
         response = requests.get(url)
         while response.status_code != 200 and number_retries < max_retries:
            time.sleep(delay)
            response = requests.get(url)
            number_retries += 1
        response.raise_for_status()
        return response.content

Any suggestions to improve it?

6
  • At minimum, you'll need to define self. It's undefined in this sample code. Commented Jan 10, 2016 at 23:14
  • I edited the code to remove it
    – Walid Saad
    Commented Jan 10, 2016 at 23:16
  • If you don't want a delay between the retries, there is built-in support by specifying your own transport adapter: stackoverflow.com/a/18190415/125629
    – John Keyes
    Commented Jan 10, 2016 at 23:19
  • Retry code can use a constant delay, but often it makes more sense to increase the retry duration on subsequent retries. There are many strategies for its. See e.g. exponential back off Commented Jan 10, 2016 at 23:20
  • @JohnKeyes the problem with that approach is that you don't know when it will retry and when it won't
    – Walid Saad
    Commented Jan 10, 2016 at 23:39

1 Answer 1

2

You could try using requests HTTPAdapter. Something like the following should work:

import time
import httplib

import requests
from requests.adapters import HTTPAdapter


class RetryHTTPAdapter(HTTPAdapter):

    SECONDS_BETWEEN_RETRIES = 5

    def __init__(self, retry_time=120, *args, **kwargs):
        self.retry_time = retry_time
        super(RetryHTTPAdapter, self).__init__(*args, **kwargs)

    def send(self, *args, **kwargs):
        for _ in range(int(self.retry_time / self.SECONDS_BETWEEN_RETRIES)):
            response = super(RetryHTTPAdapter, self).send(*args, **kwargs)
            if response.status_code == httplib.OK:
                break
            time.sleep(self.SECONDS_BETWEEN_RETRIES)
        return response

s = requests.Session()
s.mount('http://', RetryHTTPAdapter(retry_time=180))
s.mount('https://', RetryHTTPAdapter(retry_time=180))

s.get('http://example.com')
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.