My function is executed with threads:
def getdata(self, page, ...):
tries = 10
for n in range(tries):
try:
...
datarALL = []
url = 'http://website/...'.format(...)
responsedata = requests.get(url, data=data, headers=self.hed, verify=False)
responsedata.raise_for_status()
if responsedata.status_code == 200: # 200 for successful call
...
if ...
break
except (ChunkedEncodingError, requests.exceptions.HTTPError) as e:
print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
if n == tries - 1:
print ("page {0} could not be imported. Max retried reached.".format(page))
os._exit(1) #One thread max retried - close all threads and
return datarALL
as follows:
with ThreadPoolExecutor(max_workers=num_of_workers) as executor:
futh = [(executor.submit(self.getdata, page,...)) for page in pages]
for data in as_completed(futh):
datarALL.extend(data.result())
print ("Finished generateing data.")
return datarALL
Sometimes I get unexpected exceptions like: ConnectionResetError: [Errno 104] Connection reset by peer
which shuts down my program. I want to change my code so that no matter which exception happens the thread will allways retry untill the if n == tries - 1:
is met. I don't want my thread to shut down due to random exception.
I read the request exceptions info page but I don't see how I can catch all the exceptions without manually listing all of them. Is there a generic way to do this?
Basicly I want something like:
except (ALL EXCEPTIONS from Requests) as e:
print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
if n == tries - 1:
print ("page {0} could not be imported. Max retried reached.".format(page))
os._exit(1) #One thread max retried - close all threads and
return datarALL
How can I do that?
EDIT: using
except Exception as e:
print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
if n == tries - 1:
print ("page {0} could not be imported. Max retried reached.".format(page))
os._exit(1) #One thread max retried - close all threads and
return datarALL
Does not catch. it gives me this:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 331, in _error_catcher
yield
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 640, in read_chunked
chunk = self._handle_chunk(amt)
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 595, in _handle_chunk
returned_chunk = self._fp._safe_read(self.chunk_left)
File "/usr/lib/python3.5/http/client.py", line 607, in _safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
The loop does not retry. The run is terminated...
EDIT2:
except requests.exceptions.RequestException as e:
print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
if n == tries - 1:
print ("page {0} could not be imported. Max retried reached.".format(page))
os._exit(1) #One thread max retried - close all threads and
return datarALL
Also does not catch the ConnectionResetError: [Errno 104] Connection reset by peer
listed above.
except ConnectionResetError: .....
?except Exception:
?