4

I wrote a script which fetches publicly available data and puts into SQLite DB using peewee. The data is available like this site.com/data/1, site.com/data/2 ... site.com/data/N. So I use a for loop and change the value of N each time.

Problem is after a while script stops working. I mean, it just stops responding (not exits). I have print statement in loop and it stops printing next numbers.

For a smaller loop range it works perfectly and for larger it stops working. I have found that upto range of 80, it just works fine.

When I force close the script, I get following on terminal. It basically something related to requests and http connections.

Traceback (most recent call last):
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

Full traceback:

Traceback (most recent call last):
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "script.py", line 71, in <module>
    main()
  File "script.py", line 56, in main
    user_data = get_script_user_data(i)
  File "script.py", line 42, in get_script_user_data
    r = requests.get(users_data_public_api.format(user_id))
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/Users/avi/Documents/code/my-app/venv/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1171, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 351, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 313, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 374, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt

I thought I am making requests too fast and request is not able to buffer it (just my assumptions). So I added sleep statement for half second. And problem still existed. Then I changed sleep to 1.5 second, worked till loop range was upto 400. But on next iteration it stopped at within range of 50-60.

Here's the full code:

import sqlite3
import datetime
import time
from datetime import date
import requests
from peewee import *

users_data_public_api = "http://api.some-site.com/user/{0}"
db = SqliteDatabase('opendata_users.db')

class OpendataUser(Model):
    user_id = IntegerField()
    fullname = CharField()
    email = CharField()
    sex = CharField(null = True)
    dob = DateField(null = True)

    class Meta:
        database = db

def initialize_db():
    db.connect()
    #db.create_tables([OpendataUser])

def deinit():
    db.close()

def get_opendata_user_data(user_id):
    """ Returns Opendata user data only if he has all the fields required """
    r = requests.get(users_data_public_api.format(user_id))
    if r.status_code == requests.codes.ok:
        user_data = r.json()['users'][0]
        if user_data['email'] != None and 'deleted' not in user_data['email']:
            if user_data['fullname'] != None and user_data['dob'] != None:
                try:
                    user_data['dob'] = datetime.datetime.strptime(user_data['dob'], "%d/%m/%Y").date()
                except ValueError:
                    user_data['dob'] = None
                return user_data

def main():
    for i in range(450, 600):
        user_data = get_opendata_user_data(i)
        if user_data:
            print(i)
            od_user = OpendataUser(user_id=user_data['user_id'], 
                            fullname=user_data['fullname'], 
                            email=user_data['email'], 
                            sex=user_data['sex'], 
                            dob=user_data['dob'])
            od_user.save()
        time.sleep(1.5)

if __name__ == '__main__':
    initialize_db()
    main()
    deinit()

So whats the reason? appreciate any help. (code is not open source, so I have modified here and there)

3
  • For me it looks like it just waiting for response from server. You can try to shorten timeout interval.
    – werewindle
    May 12, 2015 at 15:23
  • I have added another print statement, after the GET request. It prints but it stops before putting the data into DB. So I think it's due to PeeWee.
    – avi
    May 12, 2015 at 15:42
  • Given the traceback, though, it appears when you halt your app that it is in the middle of performing a HTTP request?
    – coleifer
    May 12, 2015 at 18:28

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.