6

I am using the Python requests library to implement retry logic. Here is a simple script I made to reproduce the problem that I am having. In the case where we run out of retries, I would like to be able to log at least one of the responses from the server to help debugging. However, it is not clear to me how to access that information. Of course, I could implement retries in some other way to accomplish my goal, but it seemed like this wasn't that much of an edge case and I would be surprised to find out that requests did not support my use case.

I have looked at the requests.exceptions.RetryError, the requests.packages.urllib3.exceptions.MaxRetryError that it wraps, and the requests.packages.urllib3.exceptions.ResponseError that that wraps all to no avail.

Am I missing something?

#!/usr/bin/env python

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
from requests.exceptions import RetryError


def main():
    retry_policy = Retry(
        total=3,
        status_forcelist=[418])

    session = requests.Session()
    session.mount('http://', HTTPAdapter(max_retries=retry_policy))

    try:
        session.get('http://httpbin.org/status/418')
    except RetryError as retry_error:
        print retry_error
        print retry_error.response is None


if __name__ == '__main__':
    main()

$ python test.py

HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: /status/418 (Caused by ResponseError('too many 418 error responses',))

True

1 Answer 1

12

Checking out the urllib3 source, I discovered that the Retry object takes a named parameter called raise_on_status. True is the default. When set to False, running into the retry limit causes a response to be returned instead of an exception being thrown. The code below shows me causing an HTTPError (containing the Response) to be thrown by using the raise_for_status method on the Response, but one could just as easily use the Response directly.

#!/usr/bin/env python

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
from requests.exceptions import HTTPError


def main():
    retry_policy = Retry(
        total=3,
        status_forcelist=[418],
        raise_on_status=False)

    session = requests.Session()
    session.mount('http://', HTTPAdapter(max_retries=retry_policy))

    try:
        response = session.get('http://httpbin.org/status/418')
        response.raise_for_status()
    except HTTPError as e:
        print e.response.status_code
        print e.response.content


if __name__ == '__main__':
    main()

$ python test.py

418

-=[ teapot ]=-

   _...._
 .'  _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
  |       ;/
  \_     _/
    `"""`
1
  • Very clear and helpful, and cute to see what httpbin has in store for us also ;)
    – nealmcb
    Commented Oct 16, 2017 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.