0

I would like to convert a bunch of CSV file to a specific .JSON file format in Python.

This is my sample CSV file:

L1-CR109 Security Counter,has been forced,2019-02-26
L1-CR109 Security Counter,has been forced,2019-02-26
L1-CR109 Security Counter,has been forced,2019-02-26
L1-CR109 Security Counter,has been forced,2019-02-26

.. and this is my desired json output:

{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "location" : "L1-CR109 Security Counter", "door_activity": "has been forced", "2019-02-26"}
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "location" : "L1-CR109 Security Counter", "door_activity": "has been forced", "2019-02-26"}
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "location" : "L1-CR109 Security Counter", "door_activity": "has been forced", "2019-02-26"}
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "location" : "L1-CR109 Security Counter", "door_activity": "has been forced", "2019-02-26"}

Currently I was able to produce the following json format result:

[{"location": "L1-CR109 Security Counter", "door_status": "has been forced", "date": "2019-02-21"}, 
{"location": "L1-CR109 Security Counter", "door_status": "has been forced", "date": "2019-02-21"}, 
{"location": "L1-CR109 Security Counter", "door_status": "has been forced", "date": "2019-02-21"}, 
{"location": "L1-CR109 Security Counter", "door_status": "has been forced", "date": "2019-02-21"}

..and this is my Python code:

def csv_to_json():
    in_file = '/Elastic Search/Converted Detection/Converted CSV'
    out_file = '/Elastic Search/Converted Detection/Converted JSON'

    for filename in os.listdir(in_file):
        print("\n")
        print("Converting " + filename + " file...")
        with open(in_file + "/" + filename, 'r') as f:
            if filename.endswith(".csv"):
                reader = csv.DictReader(f, fieldnames=("location", "door_status", "date"))
                out = json.dumps([row for row in reader])

                text_file = open(out_file + r'/{}.json'.format(filename[:-4]), 'w')
                text_file.write(out + "\n")

I have tried to search for a solution but to no avail. May I know what am I missing in the code? Again can I seek advice on why Elastic Search only allows my desired json output format with the index instead of the normal python format?

1

2 Answers 2

1

This is one way to do it. Note - you hadn't given your date field a name so I've done that, to make it valid json).

import json
import csv
import sys
from collections import OrderedDict

index_line = { "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
with open('input.csv', 'r') as infile, open('outfile.json', 'w+') as outfile:

    inreader = csv.reader(infile, delimiter=',', quotechar='"')

    for line in inreader:
        document = OrderedDict()
        document['location'] = line[0]
        document['door_activity'] = line[1]
        document['date'] = line[2]
        json.dump(index_line, outfile)
        outfile.write("\n")
        json.dump(document, outfile)
        outfile.write("\n")

sys.exit()
1

Here's a version with the Python pandas package:

import json
import pandas as pd

in_file = '/Elastic Search/Converted Detection/Converted CSV'
out_file = '/Elastic Search/Converted Detection/Converted JSON'
index_line = '{"index": {"_index": "test", "_type": "_doc", "_id": "1"}}\n'

Reading:

df = pd.read_csv(in_file)

Or directly from a string:

text = "L1-CR109 Security Counter,has been forced,2019-02-26\n"*4
df = pd.read_csv(StringIO(text),header=None)

Now write the desired format (note that I've added 'date' so it's a valid JSON):

with open('outfile.json', 'w+') as outfile:
    for row in df.to_dict('records'):
       data = json.dumps(dict(list(zip(title,row.values()))))
       outfile.write(index_line+data)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.