38

I would like to convert a list into JSON array. I'm looking at jq for this but the examples are mostly about parsing JSON (not creating it). It would be nice to know proper escaping will occur. My list is single line elements so the new line will probably be the best delimiter.

2
  • 1
    BTW, the -1 wasn't me, but if I had to guess as to why it's there, it's probably the "please do my work for me" scent this (otherwise useful and worthy, hence my answer) question gives off. Showing your work and issues you've hit trying to solve this yourself would avoid that. Commented Oct 9, 2014 at 21:53
  • thank you, yes I did google around. I was looking for a small one-liner maybe something I missed in the jq man pages. Thanks for the feedback, that up-vote was me.
    – jcalfee314
    Commented Oct 9, 2014 at 23:42

5 Answers 5

50

I was also trying to convert a bunch of lines into a JSON array, and was at a standstill until I realized that -s was the only way I could handle more than one line at a time in the jq expression, even if that meant I'd have to parse the newlines manually.

jq -R -s -c 'split("\n")' < just_lines.txt
  • -R to read raw input
  • -s to read all input as a single string
  • -c to not pretty print the output

Easy peasy.

Edit: I'm on jq ≥ 1.4, which is apparently when the split built-in was introduced.

9
  • Looks easy .. I can't verify though. I get an error on ubuntu jq version 1.3 error: split is not defined
    – jcalfee314
    Commented Jan 18, 2015 at 16:18
  • 1
    Unfortunately, the jq documentation doesn't say when each built-in function was introduced, nor does jq --version tell me what version I've got, but it's at least 1.4, installed with brew install jq --HEAD. Good point, though, I'll edit the answer.
    – chbrown
    Commented Jan 18, 2015 at 20:15
  • 6
    I like this answer except that for a multi-line output that has a trailing newline(\n), the split built-in function will yield an extra empty string item in the array :( Anyway to fix that?
    – Devy
    Commented Oct 21, 2016 at 15:46
  • 3
    My current hack is to repipe it into jq to use its slicing feature ¯_(ツ)_/¯ echo $(jq -R -s -c 'split("\n")' < urls.txt) | jq '.[:-1]' Commented Feb 18, 2018 at 6:03
  • 3
    Consider 'split("\n") | map(select(length > 0))' to remove empty lines, too.
    – Justin W.
    Commented Nov 5, 2019 at 21:11
32

--raw-input, then --slurp

Just summarizing what the others have said in a hopefully quicker to understand form:

cat /etc/hosts  | jq  --raw-input .  | jq --slurp .

will return you:

[
  "fe00::0 ip6-localnet",
  "ff00::0 ip6-mcastprefix",
  "ff02::1 ip6-allnodes",
  "ff02::2 ip6-allrouters"
]

Explanation

 --raw-input/-R:

       Don´t parse the input as JSON. Instead, each line of text is passed
       to  the  filter  as  a  string.  If combined with --slurp, then the
       entire input is passed to the filter as a single long string.

 --slurp/-s:

       Instead of running the filter for each JSON object  in  the  input,
       read  the entire input stream into a large array and run the filter
       just once.
9

You can also use jq -R . to format each line as a JSON string and then jq -s (--slurp) to create an array for the input lines after parsing them as JSON:

$ printf %s\\n aa bb|jq -R .|jq -s .
[
  "aa",
  "bb"
]

The method in chbrown's answer adds an empty element to the end if the input ends with a linefeed, but you can use printf %s "$(cat)" to remove trailing linefeeds:

$ printf %s\\n aa bb|jq -R -s 'split("\n")'
[
  "aa",
  "bb",
  ""
]
$ printf %s\\n aa bb|printf %s "$(cat)"|jq -R -s 'split("\n")'
[
  "aa",
  "bb"
]

If the input lines don't contain ASCII control characters (which have to be escaped in strings in valid JSON), you can use sed:

$ printf %s\\n aa bb|sed 's/["\]/\\&/g;s/.*/"&"/;1s/^/[/;$s/$/]/;$!s/$/,/'
["aa",
"bb"]
2
  • When pipeing from jq to jq pretty printing is not necessary: jq -cR . | jq -s .
    – ceving
    Commented Oct 27, 2016 at 10:00
  • 1
    Rather than slurping the inputs in a separate call, just throw the raw inputs into an array. jq -Rn '[inputs]' input.json. These are both equivalent: jq -s '.' input.json and jq -n '[inputs]' input.json Commented Aug 10, 2017 at 3:18
9

Update: If your jq has inputs you can simply write:

jq -nR [inputs] /etc/hosts 

to produce a JSON array of strings. This avoids having to read the text file as a whole.

2
  • Adding -n makes it not skip the first line – it avoids consuming the first input separately.
    – aredridel
    Commented Jan 3, 2020 at 3:47
  • This also gets around the issue with an empty element if the input ends with a newline character. Commented Apr 2, 2022 at 2:11
6

I found in the man page for jq and through experimentation what seems to me to be a simpler answer.

$ cat test_file.txt | jq -Rsc '. / "\n" - [""]'
["aa","bb"]

The -R is to read without trying to parse json, the -s says to read all of the input as one string, and the -c is for one-line output - not necessary, but it's what I was looking for.

Then in the string I pass to jq, the '.' says take the input as it is. The '/ \n' says to divide the string (split it) on newlines. The '- [""]' says to remove from the resulting array any empty strings (resulting from an extra newline at the end).

It's one line and without any complicated constructs, using just simple built in jq features.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.