129

Can anyone recommend a Unix (choose your flavor) JSON parser that could be used to introspect values from a JSON response in a pipeline?

5

10 Answers 10

235

I prefer python -m json.tool which seems to be available per default on most *nix operating systems per default.

$ echo '{"foo":1, "bar":2}' | python -m json.tool
{
    "bar": 2, 
    "foo": 1
}

Note: Depending on your version of python, all keys might get sorted alphabetically, which can or can not be a good thing. With python 2 it was the default to sort the keys, while in python 3.5+ they are no longer sorted automatically, but you have the option to sort by key explicitly:

$ echo '{"foo":1, "bar":2}' | python3 -m json.tool --sort-keys
{
    "bar": 2, 
    "foo": 1
}
7
  • 5
    Underrated answer. This is a nice command-line alternative if the goal is to validate a given JSON file as containing valid JSON. Sep 11, 2012 at 23:19
  • 11
    this answer didn't describe how to inspect values of specified key.
    – Colin Su
    May 9, 2014 at 7:23
  • 8
    @ColinSu but that was also not the original question. json.tool is just a short hand to pretty print json. If you need to extract/manipulate json data in a shell script, I would use jq which is pure awesome at what is does...
    – muhqu
    May 9, 2014 at 8:04
  • 2
    IMMO this is an incorrect answer because python's json.tool only does two things: validate and pretty-print json. It does NOT introspect values in the json like jq does.
    – Devy
    Aug 23, 2016 at 14:58
  • 1
    @AdityaSriram good to know! …will add this info to the answer.
    – muhqu
    Aug 6, 2020 at 13:43
149

If you're looking for a portable C compiled tool:

https://stedolan.github.io/jq/

From the website:

jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

jq can mangle the data format that you have into the one that you want with very little effort, and the program to do so is often shorter and simpler than you’d expect.

Tutorial: https://stedolan.github.io/jq/tutorial/
Manual: https://stedolan.github.io/jq/manual/
Download: https://stedolan.github.io/jq/download/

3
  • 22
    Best answer in here imo. No heavy dependencies, small, powerful, good documentation and a breeze to try it out. Thanks a lot for suggesting this!
    – FrozenCow
    Oct 6, 2013 at 0:33
  • On Ubuntu/Debian you can just apt install jq. Dec 11, 2017 at 17:35
  • 2
    I asked this many-a-moon ago, and have learned to love jq.
    – Jé Queue
    Oct 17, 2019 at 19:21
62

I have created a module specifically designed for command-line JSON manipulation:

https://github.com/ddopson/underscore-cli

  • FLEXIBLE - THE "swiss-army-knife" tool for processing JSON data - can be used as a simple pretty-printer, or as a full-powered Javascript command-line
  • POWERFUL - Exposes the full power and functionality of underscore.js (plus underscore.string)
  • SIMPLE - Makes it simple to write JS one-liners similar to using "perl -pe"
  • CHAINED - Multiple command invokations can be chained together to create a data processing pipeline
  • MULTI-FORMAT - Rich support for input / output formats - pretty-printing, strict JSON, etc [coming soon]
  • DOCUMENTED - Excellent command-line documentation with multiple examples for every command

It allows you to do powerful things really easily:

cat earthporn.json | underscore select '.data .title'
# [ 'Fjaðrárgljúfur canyon, Iceland [OC] [683x1024]',
#   'New town, Edinburgh, Scotland [4320 x 3240]',
#   'Sunrise in Bryce Canyon, UT [1120x700] [OC]',
# ...
#   'Kariega Game Reserve, South Africa [3584x2688]',
#   'Valle de la Luna, Chile [OS] [1024x683]',
#   'Frosted trees after a snowstorm in Laax, Switzerland [OC] [1072x712]' ]

cat earthporn.json | underscore select '.data .title' | underscore count
# 25

underscore map --data '[1, 2, 3, 4]' 'value+1'
# prints: [ 2, 3, 4, 5 ]

underscore map --data '{"a": [1, 4], "b": [2, 8]}' '_.max(value)'
# [ 4, 8 ]

echo '{"foo":1, "bar":2}' | underscore map -q 'console.log("key = ", key)'
# key = foo
# key = bar

underscore pluck --data "[{name : 'moe', age : 40}, {name : 'larry', age : 50}, {name : 'curly', age : 60}]" name
# [ 'moe', 'larry', 'curly' ]

underscore keys --data '{name : "larry", age : 50}'
# [ 'name', 'age' ]

underscore reduce --data '[1, 2, 3, 4]' 'total+value'
# 10

And it has one of the best "smart-whitespace" JSON formatters available:

If you have any feature requests, comment on this post or add an issue in github. I'd be glad to prioritize features that are needed by members of the community.

5
  • Awesome! But, is it possible to run console commands on JSON data? For example: given a JSON file with an URL array, wget every URL. Sep 27, 2013 at 11:44
  • @CamiloMartin - the easiest way to do that is to print out the URLs, one URL per line, and then run that through xargs or GNU parallel. Sep 27, 2013 at 16:48
  • @DaveDopson Can I use underscore for parsing nested json having nested objects and arrays?
    – user227666
    Feb 20, 2014 at 9:47
  • 1
    @user227666 - sure. JSON supports nesting many levels of objects. Or you might mean JSON that has a string which encodes further JSON. Which also works, but requires just a bit of munging. Feb 25, 2014 at 20:11
  • @DaveDopson Does underscore support "contains" a "pattern", ie. for a specific "key", the possible set of (case-insenstitive) values ? I tried "jq" with match, but it doesn't work. Also posted my complete use-case here - stackoverflow.com/questions/25463196/…
    – ekta
    Aug 23, 2014 at 17:29
19

You can use this command-line parser (which you could put into a bash alias if you like), using modules built into the Perl core:

perl -MData::Dumper -MJSON::PP=from_json -ne'print Dumper(from_json($_))'
3
  • 1
    I am confused by the output of this. The output includes fat arrows (=>) between keys and values. This isn't JSON. Feb 18, 2012 at 18:34
  • 7
    @landon: no, the input is JSON, and the output is a native Perl data structure, which you can then go on to further manipulate if needed. The point of this one-liner is it produces data that is much easier to read.
    – Ether
    Apr 19, 2012 at 20:12
  • 1
    If you want a JSON output, you can use this Perl one-liner: perl -e "use JSON; print to_json( decode_json(<>), { pretty => 1 } )" Dec 24, 2018 at 1:39
14

Checkout TickTick.

It's a true Bash JSON parser.

#!/bin/bash
. /path/to/ticktick.sh

# File
DATA=`cat data.json`
# cURL
#DATA=`curl http://foobar3000.com/echo/request.json`

tickParse "$DATA"

echo ``pathname``
echo ``headers["user-agent"]``
1
  • Gotta love shell-level tools :)
    – Jé Queue
    Feb 22, 2012 at 22:01
14

There is also JSON command line processing toolkit if you happen to have node.js and npm in your stack.

And another "json" command for massaging JSON on your Unix command line.

And here are the other alternatives:


Related: Command line tool for parsing JSON input for Unix?

3
  • Easy to install, on Ubuntu: sudo apt-get install python-pip && sudo pip install jsonpipe May 22, 2012 at 11:51
  • @divideandconquer.se Sorry but you install this tool using npm with npm install json.
    – gitaarik
    Jan 26, 2014 at 15:13
  • @rednaw Unfortunately, the NPM package json seems to be taken over by a completely different package now.
    – Brad
    Sep 11, 2014 at 18:30
8

Anyone mentioned Jshon or JSON.sh?

https://github.com/keenerd/jshon

pipe json to it, and it traverses the json objects and prints out the path to the current object (as a JSON array) and then the object, without whitespace.

http://kmkeen.com/jshon/
Jshon loads json text from stdin, performs actions, then displays the last action on stdout and also was made to be part of the usual text processing pipeline.

1
  • Example usage on OSX: brew install jshon, cat *.json | jshon
    – kenorb
    Aug 29, 2014 at 9:58
2

You could try jsawk as suggested in this answer.

Really you could whip up a quick python script to do this though.

0
1

For Bash/Python, here is a basic wrapper around python's simplejson:

json_parser() {
    local jsonfile="my_json_file.json"
    local tc="import simplejson,sys; myjsonstr=sys.stdin.read(); "`
            `"myjson=simplejson.loads(myjsonstr);"
    # Build python print command based on $@
    local printcmd="print myjson"
    for (( argn=1; argn<=$#; argn++ )); do
        printcmd="$printcmd['${!argn}']"
    done
    local result=$(python -c "$tc $printcmd.keys()" <$jsonfile 2>/dev/null \
        || python -c "$tc $printcmd" <$jsonfile 2>/dev/null)
    # For returning space-separated values
    echo $result|sed -e "s/[]|[|,|']//g"
    #echo $result 
}

It really only handles the nested-dictionary style of data, but it works for what I needed, and is useful for walking through the json. It could probably be adapted to taste.

Anyway, something homegrown for those not wanting to source in yet another external dependency. Except for python, of course.

Ex. json_parser {field1} {field2} would run print myjson['{field1}']['{field2}'], yielding either the keys or the values associated with {field2}, space-separated.

0

I just made jkid which is a small command-line json explorer that I made to easily explore big json objects. Objects can be explored "transversally" and a "preview" option is there to avoid console overflow.

$  echo '{"john":{"size":20, "eyes":"green"}, "bob":{"size":30, "eyes":"brown"}}' > test3.json
$  jkid . eyes test3.json 
object[.]["eyes"]
{
  "bob": "brown", 
  "john": "green"
}
1
  • How can I install jkid in mac?
    – user227666
    Feb 20, 2014 at 9:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.