try:
r = requests.get(url, params={'s': thing})
except requests.ConnectionError, e:
print(e)
Is this correct? Is there a better way to structure this? Will this cover all my bases?
Have a look at the Requests exception docs. In short:
In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a
ConnectionError
exception.In the event of the rare invalid HTTP response, Requests will raise an
HTTPError
exception.If a request times out, a
Timeout
exception is raised.If a request exceeds the configured number of maximum redirections, a
TooManyRedirects
exception is raised.All exceptions that Requests explicitly raises inherit from
requests.exceptions.RequestException
.
To answer your question, what you show will not cover all of your bases. You'll only catch connection-related errors, not ones that time out.
What to do when you catch the exception is really up to the design of your script/program. Is it acceptable to exit? Can you go on and try again? If the error is catastrophic and you can't go on, then yes, you may abort your program by raising SystemExit (a nice way to both print an error and call sys.exit
).
You can either catch the base-class exception, which will handle all cases:
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.RequestException as e: # This is the correct syntax
raise SystemExit(e)
Or you can catch them separately and do different things.
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.Timeout:
# Maybe set up for a retry, or continue in a retry loop
except requests.exceptions.TooManyRedirects:
# Tell the user their URL was bad and try a different one
except requests.exceptions.RequestException as e:
# catastrophic error. bail.
raise SystemExit(e)
As Christian pointed out:
If you want http errors (e.g. 401 Unauthorized) to raise exceptions, you can call
Response.raise_for_status
. That will raise anHTTPError
, if the response was an http error.
An example:
try:
r = requests.get('http://www.google.com/nothere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
raise SystemExit(err)
Will print:
404 Client Error: Not Found for url: http://www.google.com/nothere
socket.timeout
exceptions if you are using a timeout: github.com/kennethreitz/requests/issues/1236
One additional suggestion to be explicit. It seems best to go from specific to general down the stack of errors to get the desired error to be caught, so the specific ones don't get masked by the general one.
url='http://www.google.com/blahblah'
try:
r = requests.get(url,timeout=3)
r.raise_for_status()
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
Http Error: 404 Client Error: Not Found for url: http://www.google.com/blahblah
vs
url='http://www.google.com/blahblah'
try:
r = requests.get(url,timeout=3)
r.raise_for_status()
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
OOps: Something Else 404 Client Error: Not Found for url: http://www.google.com/blahblah
Max retries exceeded with url:
? I've add all exceptions to the exception list but it is still not handled.
Exception object also contains original response e.response
, that could be useful if need to see error body in response from the server. For example:
try:
r = requests.post('somerestapi.com/post-here', data={'birthday': '9/9/3999'})
r.raise_for_status()
except requests.exceptions.HTTPError as e:
print (e.response.text)
Here's a generic way to do things which at least means that you don't have to surround each and every requests
call with try ... except
:
def requests_call(method, url, **kwargs):
# see the docs: if you set no timeout the call never times out! A tuple means "max
# connect time" and "max read time"
DEFAULT_REQUESTS_TIMEOUT = (5, 15) # for example
# but the coder can specify their own of course:
if 'timeout' not in kwargs:
kwargs['timeout'] = DEFAULT_REQUESTS_TIMEOUT
try:
response = requests.request(method, url, **kwargs)
except BaseException as exception:
# anticipate giant data string: curtail for logging purposes
if 'data' in kwargs and len(kwargs['data']) > 500:
kwargs['data'] = f'{kwargs["data"][:500]}...'
logger.exception(f'method |{method}|\nurl {url}\nkwargs {kwargs}')
raw_tb = traceback.extract_stack()
msg = 'Stack trace:\n' + ''.join(traceback.format_list(raw_tb[:-1]))
logger.error(msg)
return (False, exception)
return (True, response)
NB
ConnectionError
which is a builtin, nothing to do with the class requests.ConnectionError
*. I assume the latter is more common in this context but have no real idea...None
returned exception, requests.RequestException
, the superclass of all the requests
exceptions (including requests.ConnectionError
), is not requests.exceptions.RequestException
"except
block both logger.exception()
and logger.error()
, with the latter printing the call stack up to the function. In fact logger.exception
shows the call stack from here to point of failure, but tells you nothing about the call stack which called this function.*I looked at the source code: requests.ConnectionError
subclasses the single class requests.RequestException
, which subclasses the single class IOError
(builtin)
**However at the bottom of this page you find "requests.exceptions.RequestException" at the time of writing (2022-02)... but it links to the above page: confusing.
Usage is very simple:
success, deliverable = requests_call('get',
f'http://localhost:9200/my_index/_search?q={search_string}')
First you check success
: if it's False
something funny has happened and deliverable
will be an exception which has to be acted on in some way depending on context. If success
is True
then deliverable
will be a Response object.
A couple of things are tedious with regard to the above: 1) how to stipulate that only one or more status codes are acceptable? 2) how to specify that the JSON dict returned must contain a certain structure of keys and perhaps subdictionaries? In the case of Elasticsearch, for example, you often have JSON objects returned and it is tedious to check that all the keys are actually present before obtaining the values for them.
This is therefore built on the above simple function.
def process_json_request(url, method='get', ok_status_codes=200, required_dict=None, **kwargs):
if required_dict != None and type(required_dict) != dict:
raise Exception(f'required_dict must be None or dict, not {type(required_dict)} ({required_dict})')
# NB `ok_status_codes` can either be a list or an int (or None: "any status code acceptable"!
ok_codes_int = isinstance(ok_status_codes, int)
ok_codes_list = isinstance(ok_status_codes, list)
if ok_status_codes != None and (not ok_codes_int) and (not ok_codes_list):
raise Exception(f'ok_status_codes must be None, list or int, ' +\
f'not {type(ok_status_codes)} ({ok_status_codes})')
success, deliverable = requests_call(method, url, **kwargs)
if not success:
deliverable.failure_reason = 'requests_call returned False: deliverable is Exception'
deliverable.failure_code = 1
return (False, deliverable)
response = deliverable
if ok_status_codes != None and ((ok_codes_list and (response.status_code not in ok_status_codes)) \
or (ok_codes_int and (response.status_code != ok_status_codes))):
response.failure_reason = f'unacceptable status code: {response.status_code}'
response.failure_code = 2
return (False, response)
try:
delivered_json_dict = response.json()
except requests.exceptions.JSONDecodeError:
response.failure_reason = f'Response body did not contain valid json'
response.failure_code = 3
return (False, response)
def check_dictionary_key_values(required_dict, comparison_dict):
all_checks_passed = True
for key, value in required_dict.items():
if key not in comparison_dict:
logger.error(f'key absent: {key}')
all_checks_passed = False
break
if type(required_dict[key]) == dict:
sub_comparison_dict = comparison_dict[key]
if type(sub_comparison_dict) != dict:
logger.error(f'key which yields subdictionary in required does not in comparison: {sub_comparison_dict}')
all_checks_passed = False
break
if not check_dictionary_key_values(required_dict[key], sub_comparison_dict):
all_checks_passed = False
break
# if a value of "None" is given for a key this means "can be anything"
elif value != None and comparison_dict[key] != value:
logger.error(f'key {key} was found as expected but value {value} was not found, instead: {comparison_dict[key]}')
all_checks_passed = False
break
return all_checks_passed
if not check_dictionary_key_values(required_dict, delivered_json_dict):
response.failure_reason = f'delivered JSON\n{json.dumps(delivered_json_dict, indent=2)}\n' +\
f' did not satisfy required_dict\n{json.dumps(required_dict, indent=2)}'
response.failure_code = 4
return (False, response)
return (True, response)
Example use 1:
required_dict = {
'found': True,
'_source': {
'es_version': None,
},
# 'cats_and_dogs': 'shanty town',
}
success, deliverable = process_json_request(f'{url_for_specific_index}/_doc/1', required_dict=required_dict)
if not success:
logger.error(f'failed to get Ldoc 1 from index... deliverable.failure_reason:\n{deliverable.failure_reason}')
...
returned_dict = deliverable.json()
es_version_from_status_doc = returned_dict['_source']['es_version']
If I uncomment the "cats_and_dogs" line it returns success==False
because this supposedly required key is missing. Conversely, if the required_dict check passes you can be sure that "_source" and "es_version" will not produce nasty KeyError
s. You also know that key "found" had the value True
.
NB the required_dict
stipulating keys and values can be a dict
nested to any depth; and a value "None" means "value can be anything but at least check that the key is present at this location in the delivered dict".
It is also possible to make the code cope with list
s of dict
s in the recursive function check_dictionary_key_values
but this makes things a bit too complicated to include here, and will arguably be a very marginal requirement.
Example use 2:
This uses a requests
method other than "get" and, because it creates a new resource, the status code should be 201, not 200. NB the parameter ok_status_codes
can either be an int
or a list
(or indeed None
, meaning "any status code").
data = {...}
headers = {'Content-type': 'application/json'}
success, deliverable = process_json_request(f'{ES_URL}{INDEX_NAME}/_doc/1',
'put', data=json.dumps(data), headers=headers, ok_status_codes=201)
if not success:
logger.error(f'... {deliverable.failure_reason}')
....