6

I am trying to merge all json files in a directory (can be many).

I know that if there are only 2 files, I can use jq -s . file1.json file2.json. But when I tried to use the a wild card like jq -s . file*.json, it fails. Any idea?

2

5 Answers 5

10

Try this:

jq -s add *.json > merged.json
1

The answer Here is the jq answer... for more see the explanation below:

s="*.json"; jq -s "`x=-1; n=$(ls $s | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";`" $s

Step by step explanation

To merge two json files using jq use this command:

jq -s ".[0] * .[1]" file1.json file2.json

Now to merge three json files:

jq -s ".[0] * .[1] * .[2]" file1.json file2.json file3.json

Let's follow the same logic and generalise for n files using wildcards. The difficult part is to generate the string .[0] * .[1] ... .[n-1].

First let's find n. n is the number of json files in the folder. so:

ls *.json | wc -l

now let's write a loop that will build the string ".[1] * .. * .[a] * .[a+1] * .. * .[n-2] *". The kernel of this string is .[a] * so:

x=-1; n=$(ls *.json | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done;

ok and now let's add the last part .[n-1]:

x=-1; n=$(ls *.json | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";

now let's pass that string to the jq command... and let's parameterise *.json so that we can replace it with whatever we want with ease:

s="*.json"; jq -s "`x=-1; n=$(ls $s | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";`" $s

et voilà!

1

My JSON files contain JSON object. jq -s add *.json did not work, but the following works

jq -s '.' *.json > merged.json
1

From what I've seen, jq -s add *.json merges objects only at the top level (i.e. not recursively), and jq -s '.' *.json creates an array with each element being one of the JSON files.

The command Ouss wrote just hung for me, so I don't know about that.

I ended up using the following, which simple enough and worked for me to perform a recursive merge of multiple JSON files:

jq -s 'reduce .[] as $obj ({}; . * $obj)' *.json
0

While waiting a jq's solution, let me address the question "... or any tool?"

Different people mean different things by "merging", thus assuming here a plain recursive JSON merging, then with jtc (another tool) the solution would be like this (for an arbitrary number of files):

  • read all the JSONs into one super JSON array
  • merge into the 1st child (the first read JSON) the rest of read JSONs
  • output the resulting 1st JSON child in the array
jtc -J / -w[0] -mi[1:] / -w[0] *.json 

to illustrate the solution, here's how it works with couple arbitrary JSON files:

bash $ jtc -tc file1.json 
{
   "days": {
      "2020-05-18": { "seconds": 60 },
      "2020-05-19": { "seconds": 30 },
      "2020-05-20": { "seconds": 1400 }
   },
   "total": { "seconds": 1490 }
}
bash $ jtc -tc file2.json 
{
   "days": {
      "2020-05-20": { "seconds": 95 },
      "2020-05-21": { "seconds": 80 },
      "2020-05-22": { "seconds": 120 }
   },
   "total": { "seconds": 295 }
}
bash $ jtc -J / -w[0] -mi[1:] / -w[0] -tc file*.json
{
   "days": {
      "2020-05-18": { "seconds": 60 },
      "2020-05-19": { "seconds": 30 },
      "2020-05-20": {
         "seconds": [ 1400, 95 ]
      },
      "2020-05-21": { "seconds": 80 },
      "2020-05-22": { "seconds": 120 }
   },
   "total": {
      "seconds": [ 1490, 295 ]
   }
}
bash $ 

-tc flag is used only to display JSONs here in a compact format

PS. It so happens that I'm the creator of jtc tool - unix JSON processing utility.

14
  • I am sorry that I did not post my question clearly. The jsons are all with the same format, just with different data. I need to run the tool on windows as our servers are all windows. Commented Jul 23, 2020 at 13:54
  • Does jtc have a windows version? I could not find it so far. Commented Jul 23, 2020 at 13:55
  • The commands of some of the "answers" from the various threads here in stackoverflow or stackexchange do not work. For example, I used command jq -s '{ attributes: map(.attributes[0]) }' *.json. But jq throws an error "This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information." Commented Jul 23, 2020 at 14:24
  • If I used the simple command "jq.exe -s . *.json", I get exactly the same error thrown by jq. Not sure if jq supports wildcard on windows platform? If I only passed 2 files, jq does work properly. Commented Jul 23, 2020 at 14:25
  • No, there's no native version of jtc for widows yet. But it's easily compilable under cygwin environment
    – Dmitry
    Commented Jul 23, 2020 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.