4

Python 3.6. I use the streamer of tweepy to get tweets. It works well. But sometimes, if I let it open for more than 24h, I have this error

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\packages\urllib3\contrib\pyopenssl.py", line 277, in recv_into
return self.connection.recv_into(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\OpenSSL\SSL.py", line 1547, in recv_into
self._raise_ssl_error(self._ssl, result)
    File "C:\ProgramData\Anaconda3\lib\site-packages\OpenSSL\SSL.py", line 1353, in _raise_ssl_error
raise WantReadError()
  OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:


Traceback (most recent call last):
 File "C:\ProgramData\Anaconda3\lib\site-packages\requests\packages\urllib3\contrib\pyopenssl.py", line 277, in recv_into
return self.connection.recv_into(*args, **kwargs)
    File "C:\ProgramData\Anaconda3\lib\site-packages\OpenSSL\SSL.py", line 1547, in recv_into
self._raise_ssl_error(self._ssl, result)
  File "C:\ProgramData\Anaconda3\lib\site-packages\OpenSSL\SSL.py", line 1370, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (10054, 'WSAECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 302, in _error_catcher
yield
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 384, in read
data = self._fp.read(amt)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 449, in read
n = self.readinto(b)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 483, in readinto
return self._readinto_chunked(b)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 578, in _readinto_chunked
chunk_left = self._get_chunk_left()
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 546, in _get_chunk_left
chunk_left = self._read_next_chunk_size()
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 506, in _read_next_chunk_size
line = self.fp.readline(_MAXLINE + 1)
  File "C:\ProgramData\Anaconda3\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\packages\urllib3\contrib\pyopenssl.py", line 293, in recv_into
return self.recv_into(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\packages\urllib3\contrib\pyopenssl.py", line 282, in recv_into
raise SocketError(str(e))
OSError: (10054, 'WSAECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner
self.run()
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
 File "twitter_aspi_v0.8.py", line 179, in _init_stream
tweepy.Stream(auth, listener).userstream()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tweepy\streaming.py", line 396, in userstream
self._start(async)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tweepy\streaming.py", line 363, in _start
self._run()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tweepy\streaming.py", line 296, in _run
raise exception
  File "C:\ProgramData\Anaconda3\lib\site-packages\tweepy\streaming.py", line 265, in _run
self._read_loop(resp)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tweepy\streaming.py", line 315, in _read_loop
line = buf.read_line().strip()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tweepy\streaming.py", line 180, in read_line
self._buffer += self._stream.read(self._chunk_size)
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 401, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "C:\ProgramData\Anaconda3\lib\contextlib.py", line 100, in __exit__
self.gen.throw(type, value, traceback)
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\packages\urllib3\response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(10054, \'WSAECONNRESET\')",)', OSError("(10054, 'WSAECONNRESET')",))

My code is pretty long and regarding the error, it seems it comes from urllib3, OpenSSL and tweepy way of accessing the Twitter API. So I could handle this with a try before launching the streamer but I would like to know if there is maybe a better fix I could do to understand and avoid this? Thanks!

8
  • Where is your code?
    – Morse
    Apr 1, 2018 at 0:52
  • Are you running this code at home or on a server? Apr 1, 2018 at 5:09
  • 4
    This looks more like a temporary connection timeout which is not handled by Tweepy, so you should just write a wrapper around the same and catch the exception and restart it. I don't think the exception can be avoided as such because you connect to an external site and sometimes it could timeout Apr 1, 2018 at 18:27
  • 2
    You should look at this docs.tweepy.org/en/v3.5.0/streaming_how_to.html#handling-errors for the error handling part and see if on_error gets called in your case when connection timeout happens Apr 1, 2018 at 19:43
  • 2
    Check limits on the site 1 2 3 there is no alternative than to restart stream connection via tweepy
    – Morse
    Apr 4, 2018 at 20:00

2 Answers 2

3
+50

This looks more like a temporary connection timeout which is not handled by Tweepy, so you should just write a wrapper around the same and catch the exception and restart it. I don't think the exception can be avoided as such because you connect to an external site and sometimes it could timeout

You should look at this http://docs.tweepy.org/en/v3.5.0/streaming_how_to.html#handling-errors for the error handling part and see if on_error gets called in your case when connection timeout happens

class MyStreamListener(tweepy.StreamListener):

    def on_error(self, status_code):
        if status_code == 420:
            #returning False in on_data disconnects the stream
            return False

If this doesn't help then use the wrapper approach

2

According to Twitter Developer documentation : rate limiting this is expected to connection reset/failure when you cross your usage limit.

Requests / 15-min window (user auth) = 900

Requests / 15-min window (app auth) = 1500

Also it clearly states as below.

If the initial reconnect attempt is unsuccessful, your client should continue attempting to reconnect using an exponential back-off pattern until it successfully reconnects.

(Update)

Regardless of how your client gets disconnected, you should configure your app to reconnect immediately. If your first reconnection attempt is unsuccessful, we recommend that your app implement an exponential back-off pattern in subsequent reconnection attempts (e.g. wait 1 second, then 2 seconds, then 4, 8, 16, etc), with some reasonable upper limit. If this upper limit is reached, you should configure your client to notify your team so that you can investigate further.

The standard (free) Twitter APIs i.e Tweepy API consist of a REST API and a Streaming API. The Streaming API provides low-latency access to Tweets. Ads API has other limits when white listed.

REST API Limit

Clients may access a theoretical maximum of 3,200 statuses via the page and count parameters for the user_timeline REST API methods. Other timeline methods have a theoretical maximum of 800 statuses. Requests for more than the limit will result in a reply with a status code of 200 and an empty result in the format requested. Twitter still maintains a database of all the Tweets sent by a user. However, to ensure performance, this limit is in place on the API calls.

Could be simple reason that users should not spam that this thing is enforced.

Solution :

You may catch exception and re-establish connection to Twitter and continue reading tweets.

There is no alternative than getting better usage allowance from Twitter as of now unfortunately.

1
  • Thanks for your answer, but as i use the streaming API the limit are not the same than the Rest API. And I already had rate limiting error with the streaming API and the error is not the same as the one i've posted
    – TmSmth
    Apr 7, 2018 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.