1

Here is my df:

text date channel sentiment product segment
0 I like the new layout 2021-08-30T18:15:22Z Snowflake predict Skills EMEA

I need to convert this to JSON output that matches the following:

[
  {
    "text": "I like the new layout",
    "date": "2021-08-30T18:15:22Z",
    "channel": "Snowflake",
    "sentiment": "predict",
    "fields": [
      {
        "field": "product",
        "value": "Skills"
      },
      {
        "field": "segment",
        "value": "EMEA"
      }
    ]
  }
]

I'm getting stuck with mapping the keys of the columns to the values in the first dict and mapping the column and row to new keys in the final dict. I've tried various options using df.groupby with .apply() but am coming up short.

Samples of what I've tried:

df.groupby(['text', 'date','channel','sentiment','product','segment']).apply(
     lambda r: r[['27cf2f]].to_dict(orient='records')).unstack('text').apply(lambda s: [
{s.index.name: idx, 'fields': value}
for idx, value in s.items()]
).to_json(orient='records')

Any and all help is appreciated!

2
  • I believe your given df has some column missing. Also you might want to add a line before table or it's not correctly shown.
    – Cookie
    Sep 27, 2021 at 23:13
  • @Cookie The df is correct. Thanks for the feedback on the markdown tho. Sep 28, 2021 at 10:55

1 Answer 1

1

One option is to use a nested list comprehension:

# Start with your example data
d = {'text': ['I like the new layout'],
     'date': ['2021-08-30T18:15:22Z'],
     'channel': ['Snowflake'],
     'sentiment': ['predict'],
     'product': ['Skills'],
     'segment': ['EMEA']}

df = pd.DataFrame(d)

# Specify field column names
fieldcols = ['product', 'segment']

# Build a dict for each group as a Series named `fields`
res = (df.groupby(['text', 'date','channel','sentiment'])
 .apply(lambda s: [{'field': field, 
                    'value': value}
                   for field in fieldcols
                   for value in s[field].values])
).rename('fields')

# Convert Series to DataFrame and then to_json
res = res.reset_index().to_json(orient='records')

# Print result
import json
print(json.dumps(json.loads(res), indent=2))

[
  {
    "text": "I like the new layout",
    "date": "2021-08-30T18:15:22Z",
    "channel": "Snowflake",
    "sentiment": "predict",
    "fields": [
      {
        "field": "product",
        "value": "Skills"
      },
      {
        "field": "segment",
        "value": "EMEA"
      }
    ]
  }
]
4
  • Thanks Peter! This works perfectly. Marked as accepted! As a follow on, how would you do this if you had multiple columns you needed in the fields array? Sep 28, 2021 at 11:26
  • Thanks Matthew! Do you mean how to extend this to work on multiple additional columns with names like 27cf2f? I would try something like a second list or dict comprehension to loop over the field-id columns, or maybe a pd.melt first to move the field ids from column names into a single column, and the field values into their own column. Sep 28, 2021 at 13:32
  • Hey Peter. Yes that's correct. I've updated the question to reflect the needed logic as I'm struggling to get the code worked out for the addtional columns. Thanks! Sep 28, 2021 at 21:00
  • Got it. I've edited my answer to work on multiple field columns. Hopefully that works! Sep 29, 2021 at 1:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.