1

For my project I have to parse two big JSON files, one is 19.7 GB and another 66.3 GB. The structure of the JSON data is too complex. First Level Dictionary and again in 2nd level there might be List or Dictionary. These are all Network Log files, I have to parse those log files and do analysis. Is converting such big JSON file to CSV is advisable?

When I am trying to convert the smaller 19.7 GB JSON file to CSV file, it is having around 2000 columns and 0.5 millions of rows. I am using Pandas to parse those data. I have not touched the bigger file 66.3 GB. Whether I am going in right direction or not? When I 'll convert that bigger file, how many columns and rows will come out, there is no idea.

Kindly suggest any other good options if exists. Or is it advisable to directly read from JSON file and apply OOPs concept over it.

I have already read these articles: article 1 from Stack Overflow and article 2 from Quora

9
  • 1
    You should probably use c instead of python for this kind of stuff.
    – vishal
    Jul 11, 2018 at 6:31
  • 1
    @debaonline4u No need to learn a new programming lang C, You can very well do this in Python, We have proccessed json with 20 million keys and much more nested than yours. Get that into Pandas dataframe first and then you can do any manipulation you want to..
    – min2bro
    Jul 11, 2018 at 6:34
  • 1
    I prefer instead of converting it to CSV use json streamer. Jul 11, 2018 at 6:34
  • The structure cannot be that complex if it can be converted to CSV. Consider using a binary format which you can memory map, a format like HDF5, or a database. Jul 11, 2018 at 6:35
  • 2

1 Answer 1

1

you might want to use dask its has similar syntax to pandas only its parallel (essentially its lots of parallel pandas datafames) and lazy (this helps with avoiding ram limitations).

you could use the read_json method and then do your calculations on the dataframe.

6
  • how much time it'll take to study dask library and work. Jul 11, 2018 at 6:45
  • if I read the whole 20 GB of data with read_json method, and convert it to a dataframe, then how much memory it's going to consume? Jul 11, 2018 at 6:47
  • 2
    for me it was about a hour and a half, they great notebooks on their website . you can also watch this video , it provides a good explanation about the library.
    – moshevi
    Jul 11, 2018 at 6:50
  • 1
    you can specify the blocksize (so the json will not be in 1 partition) do the calculations and then for example save in csv files. because it is lazy at no time will the entire file be in memory.
    – moshevi
    Jul 11, 2018 at 7:08
  • hi @moshevi, I am currently struggling in loading big files of Json lines in Pandas for my experiments. I am playing with the chunksize parameter, and in particular I noticed that bigger the chunks faster the parsing / greater the memory output. So far I reached a good balance with 10k chunks a decent parsing time and a memory usage up to 5.7 times the original file ( this proportion stands so far for 280Mb input files and 4.4 Gb). Do you think Dask can do any good here ? Jul 16, 2018 at 11:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.