2

I have below needs for retry:

My Python script uses REQUESTS module to call API & parse JSON response.

lately the API returns incomplete data like below

Incomplete data:

  {
    offerInfo: {
      siteID: "001",
      language: "en_US",
      currency: "USD"
    },
    userInfo: {
      persona: {
        personaType: "OTHERS"
      },
      userId: "111"
    },
    offers: { }
  }

Complete data:

  {
    offerInfo: {
      siteID: "001",
      language: "en_US",
      currency: "USD"
    },
    userInfo: {
      persona: {
        personaType: "OTHERS"
      },
      userId: "111"
    },
    offers: { 
      Flight: [
        {
          offerDateRange: {
            travelStartDate: [2016, 5, 7],
            travelEndDate: [2016, 5, 11]
          }
       }
    }
  }

A complete response should have ['offers']['Flight'] keys in it ,if not my code should try to call the API N times before it gives up & go to the next API URL.

I am trying with below code with session,but not sure how can I add the check in it ['offers']['Flight']

Note: There is no HTTP error while getting incomplete data returned from API ,so I have to rely on keys ['offers']['Flight']

session = requests.Session()
session.mount("http://", requests.adapters.HTTPAdapter(max_retries=2))
session.mount("https://", requests.adapters.HTTPAdapter(max_retries=2))
try:
    response = session.get(url=line,timeout=(connect_timeout,read_timeout),verify=False)
1
  • 3
    Add a loop? while True: ... and break when you have correct data or you tried too many times. Commented Apr 26, 2016 at 20:48

2 Answers 2

2

A while loop plus a count as suggested in the comments is the simple answer and may well be good enough.

If instead you'd like to use a module catered to this sort of thing, backoff is a library for decorating functions for retry with configurable backoff behavior. In particular, the on_predicate decorator allows you define criteria for retry based on the return value of a specified function.

import backoff

def has_flight_offer(dct):
    return 'Flight' in dct.get('offers')

@backoff.on_predicate(backoff.constant, has_flight_offer, max_tries=3, interval=1)
def get_offers(session, url))
    response = session.get(url, verify=False)
    return response.json()

It also handles retrying on exceptions and it plays well with requests, so you can use it to handle requests exceptions while you're at it with similar decoration.

@backoff.on_predicate(backoff.constant, has_flight_offer, max_tries=3, interval=1)
@backoff.on_exception(backoff.expo, requests.exceptions.RequestException, max_tries=3)
def get_offers(session, url))
    response = session.get(url, verify=False)
    return response.json()
0

You can use tenacity.

Doc: https://tenacity.readthedocs.io/en/latest/

It can log before or after

import logging

logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

logger = logging.getLogger(__name__)

@retry(stop=stop_after_attempt(3), before=before_log(logger, logging.DEBUG))
def raise_my_exception():
    raise MyException("Fail")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.