3

I'm trying to set up a for loop to pull in elected representatives' data for about 600,000 postal codes. The base URL stays the same, and the only part that changes is the postal code.

I'd ideally like to create a list of all the postal codes, and then use requests.get to pull in data for all the postal codes in my list. I came up with this code below, but it's only pulling in data for the last postal code in my list. I'm not really sure why this is happening and am a python beginner - so any help would be appreciated!


#loop test 
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

for i in range(len(postcodes)):
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcodes[i]))

data1=json.loads(rr.text)

data1
1
  • Because in each iteration you are overriding the value of data1. You need to save the value of data1 in somewhere (for example in lists).
    – cengineer
    Nov 30, 2018 at 13:53

3 Answers 3

5

Your code doesnt works because it overwrites data1.

Try this:

#loop test 
responses = list() # stores responses for postal codes
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

for postcode in postcodes:
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcode))
   data=json.loads(rr.text)
   responses.append(data)

Your responses now are saved in responses list.

Tips:
You can iterate over list without using index.

2
  • 2
    THANK YOU!!! Of course, duh I didn't need an index. Can't believe I forgot to append results.
    – M.Ai
    Nov 30, 2018 at 14:05
  • This answer does not contains status code check, which can bring you nice headache if it will fail after a while and none of your responses are being stored. If you will try to json.loads something that is not json it will fail on JSONDecodeError. Also if this dataset is HUGE, dont use lists, but store it in file locally, per postcode and then merge it. Nov 30, 2018 at 14:26
1

You are overwriting data1 variable every iteration, that is why you end up with only the last one, you need to store it differently.

Example:

postcodes =['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']
results = []

for postcode in postcodes:
    res = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcode))

    if res.status_code == 200:
        results.append(res.json())
    else: 
        print("Request to {} failed".format(postcode))
1

you are just viewing the last response.

#loop test 
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

api_data = dict()

for i in postcodes:
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(i))

   data = json.loads(rr.text)
   api_data.update({i: data})   
   # or print(data)

print(api_data)

here I've added all the response to a dict, with key as postal code and value as the response.

3
  • Thanks a ton! Just needs to be api_data.update(), I think?
    – M.Ai
    Nov 30, 2018 at 14:06
  • 1
    api_data.update() means that you are putting a key-value pair to api_data dictionary, that's a basic operation of dictionaries, run my code for better understanding
    – P.hunter
    Nov 30, 2018 at 14:14
  • He was refering to a typo, this code wont run since you are using api_data and then api_dict Nov 30, 2018 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.