2

I have a json file with the below structure:

[{
"field1": "first",
"field2": "d",
"id": 35,
"features": [
    {
        "feature_id": 2,
        "value": 6
    },
    {
        "feature_id": 3,
        "value": 8.5
    },
    {
      "feature_id":5,
      "value":6.7
    },
    {
    "feature_id":10,
    "value": 3.4
    }
  ],
  "time": "2018-11-17"
},
{
"field1": "second",
"field2": "b",
"id": 36,
"features": [
    {
        "feature_id": 3,
        "value": 5.4
    },
    {
        "feature_id": 10,
        "value": 9.5
    },

  ],
  "time": "2018-11-17"
}]

I can change this to Pandas Dataframe

import json
import pandas as pd
with open(file) as json_data:
 data = json.load(json_data)

df=pd.DataFrame(data)

but one column has a nested dictionary in a list and therefore the features column contains the column with the list of dictionaries. I want to flatten my whole data so the final table should look like this. Appreciate any help?

Final_dataframe

2
  • Your json file is not valid json syntax, so that seems to likely be a reason that it doesn't work. If you have a list of items it should be inside [ ] separated by commas for json. Also please show the current format of the data/DataFrame and why that is not correct.
    – Karl
    Nov 17, 2018 at 19:24
  • Sure, I'll edit it to be inside a list.
    – John
    Nov 18, 2018 at 2:18

2 Answers 2

1

For flatten JSON object with nested keys into single Dict, use the below function.

def flatten_json(nested_json):
"""
    Flatten json object with nested keys into a single level.
    Args:
        nested_json: A nested json object.
    Returns:
        The flattened json object if successful, None otherwise.
"""
out = {}

def flatten(x, name=''):
    if type(x) is dict:
        for a in x:
            flatten(x[a], name + a + '_')
    elif type(x) is list:
        i = 0
        for a in x:
            flatten(a, name + str(i) + '_')
            i += 1
    else:
        out[name[:-1]] = x

flatten(nested_json)
return out

Hope this function will help you.

0

Json is chnaged with [] for valid:

data = [{
"field1": "first",
"field2": "d",
"id": 35,
"features": [
    {
        "feature_id": 2,
        "value": 6
    },
    {
        "feature_id": 3,
        "value": 8.5
    },
    {
      "feature_id":5,
      "value":6.7
    },
    {
    "feature_id":10,
    "value": 3.4
    }
  ],
  "time": "2018-11-17"
},
{
"field1": "second",
"field2": "b",
"id": 36,
"features": [
    {
        "feature_id": 3,
        "value": 5.4
    },
    {
        "feature_id": 10,
        "value": 9.5
    },

  ],
  "time": "2018-11-17"
}]

Then loop each item and for features create new items of dictioanries, last pass is to DataFrame contructor:

L = []
for x in data:
    d = {}
    for k, v in x.items():
        if k == 'features':
            for y in v:
                d[f"feature_id_{y['feature_id']}"] = y['value']
        else:
            d[k] = v
    L.append(d)

df = pd.DataFrame(L)
print (df)
   feature_id_10  feature_id_2  feature_id_3  feature_id_5  field1 field2  id  \
0            3.4           6.0           8.5           6.7   first      d  35   
1            9.5           NaN           5.4           NaN  second      b  36   

         time  
0  2018-11-17  
1  2018-11-17  
2
  • Thanks @jezrael, can you explain the meaning of f in this line of the code before "feature_id_{y['feature_id']}" d[f"feature_id_{y['feature_id']}"] = y['value']
    – John
    Nov 18, 2018 at 2:20
  • @John - it create dictionary by f-strings - each value of nested dictionaries in list convert to new one.
    – jezrael
    Nov 18, 2018 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.