18

I have the following JSON data:

{"id":"111","case":"Y","custom":{"speech invoked":"no","input method":"hard","session ID":"420"}}

How can I convert it to CSV format using jq so my result looks like this?

id,case,session Id,speech invoked,input method

111,Y,420,no,hard

I tried the following, but it didn't work:

{(.id),(.case),(.custom."session Id"),(.custom."speech invoked"),(.custom."input method")}

If not possible any perl or shell solution is appreciated.

5 Answers 5

22

Building upon Joe Harris' answer, you can use the @csv filter so that strings are properly quoted and escaped when necessary :

jq -r '[.case, .custom."speech invoked", .custom."input method"] | @csv'
1
  • Looks better than the join but always produces double quotes around the values. Commented Nov 24, 2017 at 10:27
18

Using perl wasn't a good solution for me but after a bit of trial and error I figured out you can do it with just jq using the join() operator.

First make an array of the output you need, then join the array elements using commas.

jq -r '[.case, .custom."speech invoked", .custom."input method"] | join(", ")'

Enjoy. :)

1
  • I left off .custom."session ID" so that people can see the join() without scrolling.
    – Joe Harris
    Commented Nov 7, 2014 at 21:31
12

Using jq, you can use this filter:

with_entries(select(.key != "custom")) + .custom
    | to_entries
    | map(.key), map(.value)
    | @csv

Just note that written this way, the "custom" properties will always be written in the end, no matter what order the properties are in.

1
  • 1
    This is cool because it gets the keys automatically without knowing them in advance and it outputs column headers. But the problem is that the headers are repeated for every row. If you have variable input you could safely output only the values but if the input is variable you'll need the headers. Stash the input and output the headers only into your file first and then append the values in a second pass. Will prevent piping the output to gzip though.
    – Joe Harris
    Commented Feb 9, 2016 at 16:46
1

Here is another solution. If data.json contains the sample data then

jq -M -s -r 'map(.+.custom|del(.custom)) | (.[0]|keys_unsorted), (.[]|[.[]]) | join(",")' data.json

will produce

id,case,speech invoked,input method,session ID
111,Y,no,hard,420
0

Using Perl and its JSON module:

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

use JSON;

my $input = << '__JSON__';
{"id":"111","case":"Y","custom":{"speech invoked":"no","input method":"hard","session ID":"420"}}
__JSON__

my $struct = decode_json($input);

my @header = grep ! ref $struct->{$_}, keys %$struct;
push @header, map keys %{ $struct->{$_} },
              grep ref $struct->{$_},
              keys %$struct;

my @row = grep ! ref, values %$struct;
push @row, map values %$_, grep ref, values %$struct;

say join ',', @header;
say join ',', @row;
4
  • I'd still want to use Text::CSV_XS to create the output just in case some of the data includes things like spaces, quotes, commas, etc.
    – Tanktalus
    Commented Aug 29, 2014 at 4:46
  • @Tanktalus: Sure. Just tweak the last two lines.
    – choroba
    Commented Aug 29, 2014 at 7:22
  • 2
    This doesn't use jq like the question asks. Commented May 30, 2016 at 8:58
  • 2
    Not sure why this solution is downvoted. jq is definitely preferred over Perl, but the question says any perl or shell solution is appreciated.
    – ramhiser
    Commented Apr 25, 2017 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.