0

I'm reading in a CSV file and output-ing a JSON file which is then processed by another application. Some of the information needed by the downstream app is not present in the CSV file and needs to be obtained programmatically from another source. Is there a way to edit/insert fields into the CSV data prior to output-ing the JSON data? I've been glassing over the Python CSV module docs and noticed the for row in reader: example. I can iterate over the known fields and then insert the new ones to build a new_row variable. I can dump new_row to the JSON file but the field names given to the original reader don't coming along for the ride so I end up with a pretty crazy new_row variable that doesn't feel like I'm saving anything. Thoughts?

#!/usr/bin/python

import urllib
import json
import sys
import csv

fieldnames = ("first_name", "last_name","phone")
reader = csv.DictReader(csvfile, fieldnames)
all_lines = list(reader)

jsonfile.write ('{"people":[')

for row in all_lines:
   jsonfile.write('{"person":')

   #address = my_get_address (...)
   new_row = ["first_name", row['first_name'], "last_name", row['last_name'], "phone", row['phone'], 'address', '123 anystreet']
   json.dump(new_row, jsonfile)

   jsonfile.write('}')
   # comma + newline for every person excep the last
   if row != all_lines[-1]:
      jsonfile.write (',\n')

jsonfile.write (']}')

JSON output:

{                                                                                                                                                                                       
   "people" : [                                                                                                                                                                         
      {                                                                                                                                                                                 
         "person" : [                                                                                                                                                                  
            "first_name",                                                                                                                                                              
            "joe",                                                                                                                                                                     
            "last_name",                                                                                                                                                               
            "smith",                                                                                                                                                                   
            "phone",                                                                                                                                                                   
            "999-555-1212",                                                                                                                                                            
            "address",                                                                                                                                                                 
            "123 anystreet"                                                                                                                                                            
         ]                                                                                                                                                                             
      },                                                                                                                                                                               
      {                                                                                                                                                                                
         "person" : [                                                                                                                                                                  
            "first_name",                                                                                                                                                              
            "mary",                                                                                                                                                                    
            "last_name",                                                                                                                                                               
            "jones",                                                                                                                                                                   
            "phone",                                                                                                                                                                   
            "888-555-1212",                                                                                                                                                            
            "address",                                                                                                                                                                 
            "123 anystreet"
         ]
      }
   ]
}
5
  • 1
    Can you show the head (first lines) of your .csv file? Mar 22, 2021 at 18:38
  • 1
    Let the json module build the whole JSON string instead of manually concentrate things. Mar 22, 2021 at 18:57
  • Reading the question assuming that you want to convert csv to json and also add a custom column to the csv. Would suggest to see this problem as two separate steps. Read csv either by csv.DictReader or pandas.read_csv(or any other method of your choice) and populate a in memory variable data while populating the the it add extra columns/ editing it you want. Once that done you can simply dump the data to a json file as second step.(geeksforgeeks.org/convert-csv-to-json-using-python) Mar 22, 2021 at 18:59
  • @RobertHarvey yep. Mondays are like that sometimes ;)
    – Nstevens
    Mar 22, 2021 at 19:16
  • @RenanLopes There is no header in the CSV file. This is the way it comes out of Oracle from what I'm told so the first line is just "mary", "jones", "888-555-1212"
    – Nstevens
    Mar 22, 2021 at 19:17

1 Answer 1

1

I'm assuming your .csv file is formatted as the following:

first_name,last_name,phone
abc,abc,123
def,def,456
ghi,ghi,789

Then you probably want to do this (using your address example):

import csv 
import json 

def csv_to_json(csvFilePath, jsonFilePath):
    jsonArray = []
      
    # Read csv file
    with open(csvFilePath, encoding='utf-8') as csvf: 
        # Load csv file data using csv library's dictionary reader
        csvReader = csv.DictReader(csvf) 

        # Convert each csv row into python dict
        for row in csvReader: 
            # Update row with new fields
            row.update({'address': '123 anystreet'})
            # Add this python dict to json array
            jsonArray.append(row)
  
    # Convert python jsonArray to JSON String and write to file
    with open(jsonFilePath, 'w', encoding='utf-8') as jsonf: 
        jsonString = json.dumps(jsonArray, indent=2)
        jsonf.write(jsonString)
          
csvFilePath = r'test.csv'
jsonFilePath = r'test.json'
csv_to_json(csvFilePath, jsonFilePath)

source

Resulting .json file:

[
  {
    "first_name": "abc",
    "last_name": "abc",
    "phone": "123",
    "address": "my_address"
  },
  {
    "first_name": "def",
    "last_name": "def",
    "phone": "456",
    "address": "my_address"
  },
  {
    "first_name": "ghi",
    "last_name": "ghi",
    "phone": "789",
    "address": "my_address"
  }
]

Edit

To format the output like your question you can change these two lines to wrap row and jsonArray this way:

jsonArray.append({'person': row})
  
jsonString = json.dumps({'people': jsonArray}, indent=2)

Now it gives:

{
  "people": [
    {
      "person": {
        "first_name": "abc",
        "last_name": "abc",
        "phone": "123",
        "address": "my_address"
      }
    },
    {
      "person": {
        "first_name": "def",
        "last_name": "def",
        "phone": "456",
        "address": "my_address"
      }
    },
    {
      "person": {
        "first_name": "ghi",
        "last_name": "ghi",
        "phone": "789",
        "address": "my_address"
      }
    }
  ]
}
1
  • row.update is a piece I didn't glean from the docs. Makes a big difference, thank you for that and the thorough answer.
    – Nstevens
    Mar 22, 2021 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.