86

How can I write a piped output that went through jq to a file in shell

Example:

curl api.example.com | jq > call.txt

won't work. Neither does

(curl api.example.com | jq) > call.txt

Help!

Edit: So doing curl api.example.com > call.txt works just fine. So it has to do with piping it to jq

13
  • What output is it?
    – iBug
    Feb 24, 2018 at 15:16
  • @iBug it should not matter what output it is, but it is JSON
    – supersize
    Feb 24, 2018 at 15:16
  • Did you verify there is any output when you don't redirect it?
    – iBug
    Feb 24, 2018 at 15:18
  • @ibug yes, it writes it to the shell
    – supersize
    Feb 24, 2018 at 15:19
  • Which OS, distribution and version do you use?
    – Cyrus
    Feb 24, 2018 at 15:23

3 Answers 3

105

Just calling jq without a filter will throw errors if stdout isn't a terminal

$ curl https://jsonplaceholder.typicode.com/posts/1 | jq > test.txt
jq - commandline JSON processor [version 1.5-1-a5b5cbe]
Usage: jq [options] <jq filter> [file...]

        jq is a tool for processing JSON inputs, applying the
        given filter to its JSON text inputs and producing the
[...]

Try jq '.' (i.e: pretty-print the input JSON):

$ curl https://jsonplaceholder.typicode.com/posts/1 | jq '.' > test.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   292  100   292    0     0   1698      0 --:--:-- --:--:-- --:--:--  1707

Note that the filter is not really optional:

From man jq:

JQ(1)                                                                                JQ(1)

NAME
       jq - Command-line JSON processor

SYNOPSIS
       jq [options...] filter [files...]

According to the tip of the master branch... your described (and my observed) behaviour is not expected...

Older versions of jq have the following: (here)

if (!program && isatty(STDOUT_FILENO) && !isatty(STDIN_FILENO))
  program = ".";

i.e: use a default filter if stdout is a TTY, and stdin is not a TTY.

This behaviour appears to be corrected in commit 5fe05367, with the following snippet of code:

if (!program && (!isatty(STDOUT_FILENO) || !isatty(STDIN_FILENO)))
  program = ".";
8
  • when I run curl api.example.com | jq it gives me a pretty printed output of the JSON I just curled. How can write this to a file without saving it to a json first.
    – supersize
    Feb 24, 2018 at 15:34
  • 5
    You must specify the filter, to pass everything through give the filter as "." - e.g: curl api.example.com | jq . > call.txt
    – Attie
    Feb 24, 2018 at 15:36
  • can you amend your question to make it work with curl? The "." doesn't work for me neither
    – supersize
    Feb 24, 2018 at 15:37
  • sorry my fault, it works fine with . or ".". But I don't understand, why can I not take the output printed to the shell. What does the . to make a difference
    – supersize
    Feb 24, 2018 at 15:38
  • 3
    . is the most basic "do nothing" filter... jq clearly has some logic to assume that if no filter is given, and the output is a PTY, then use the filter .... If the output is not a PTY (e.g: it's a file), then this assumption is not made.
    – Attie
    Feb 24, 2018 at 15:39
26

My incantation:

$ cat config.json

{
    "ProgramSettings":
    {
        "version": "1.0"
    },
    "ProgramSecrets":
    {
        "AWS_ACCESS_KEY_ID": "",
        "AWS_SECRET_ACCESS_KEY": ""
    }
}

assume you want remove object 'ProgramSecrets' from JSON file:

$ echo $(cat config.json | jq 'del(.ProgramSecrets)') > config.json
$ cat config.json
{ "ProgramSettings": { "version": "1.0" } }
8
  • 8
    although interesting to know, this does not provide any answer to the above question.
    – supersize
    Oct 31, 2018 at 9:39
  • 5
    This will likely also stamp on config.json ... i.e: > config.json will truncate it before cat config.json can get a chance to read it.
    – Attie
    Apr 17, 2019 at 13:10
  • 4
    @simesy - you might like to take a look at sponge!
    – Attie
    Sep 18, 2019 at 11:00
  • 1
    @supersize This answer actually works for me, because it captures the jq output and echos it to the file, where redirecting the jq output directly to the file was failing May 15, 2020 at 19:44
  • 3
    I have tried like this and it worked. cat manifest.json | jq ".version |= \"${VERSION}\"" --indent 4 | tee manifest.json
    – ssi-anik
    Mar 25, 2021 at 1:00
5

Just try:

curl api.example.com | jq '.' > call.txt

It works fine for me.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.