267

I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).

#!/bin/sh

#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="[email protected]") | .id')
echo "$projectID"

[email protected]

#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
2
  • 9
    A related issue: passing bash variable to jq filter has the slightly different syntax jq -r --arg var "$var" '.[$var]' stackoverflow.com/questions/34745451/…
    – enharmonic
    Commented Feb 19, 2020 at 18:11
  • EMAILIDis between single quotes and does not get expanded. Write it as '"$EMAILID"' to take it out of the single quotes. Commented Mar 24, 2023 at 8:53

12 Answers 12

402

Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):

   projectID=$(jq -r --arg EMAILID "$EMAILID" '
        .resource[]
        | select(.username==$EMAILID) 
        | .id' file.json)

Postscript

For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:

[email protected]  # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'

The output is a JSON string:

"[email protected]"

shell arrays

Unfortunately, shell arrays are a different kettle of fish. Here are two SO resources regarding the ingestion of such arrays:

JQ - create JSON array using bash array with space

Convert bash array to json array and insert to file using jq

10
  • 6
    This is the only 100%-safe answer; it lets jq properly create the filter using the value, rather than using bash to create a string that jq interprets as a filter. (Consider what happens if the value of EMAILID contains a ).)
    – chepner
    Commented Oct 16, 2016 at 2:35
  • 2
    My use case, it works! function n2 { termux-contact-list |jq -r --arg v1 "$1" '.[] | select(.name==$v1)|.number' } Calling: n2 Name1
    – Timo
    Commented Aug 19, 2018 at 7:12
  • 6
    fyi, jq's env function is changed between jq 1.4 and 1.5+, so references to env.EMAILID (in this answer's example) do not work in jq 1.4, thus recommending using the --arg EMAILID "$EMAILID" construct if you need to use jq 1.4 . Hope this helps; just took me a day+ to figure this out for myself ;)
    – m0j0hn
    Commented Oct 4, 2018 at 20:15
  • 2
    In my case I had to use parenthesis around the variable name, and since it was inside a select I had to escape it too, like this jq --arg var "${SHELL_VAR}" '.UserPools[] | select(.Name == "\($var)-some-string")'
    – Federico
    Commented May 23, 2019 at 2:24
  • 5
    Arguments passed using --arg will be passed as strings. If you want to pass data of a different JSON type, use --argjson, which will parse the string as JSON, e.g., --argjson myObj '{"a": [1, 2]}' Commented Feb 17, 2020 at 16:59
56

I resolved this issue by escaping the inner double quotes

projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
1
  • 2
    I am using this because --arg doesn't play nice with -c
    – dimisjim
    Commented Apr 30, 2020 at 12:56
45

Little unrelated but I will still put it here, For other practical purposes shell variables can be used as -

value=10
jq  '."key" = "'"$value"'"' file.json
1
  • Thank you very much, it worked. I was unable to insert the BASH variable in jq query. However this method of string concatenation worked Commented Oct 6, 2023 at 12:03
29

Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:

.items[] | select(.name=="string")

in bash you could do

EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')

essentially escaping the quotes and passing it on to jq

0
20

It's a quote issue, you need :

projectID=$(
  cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id"
)

If you put single quotes to delimit the main string, the shell takes $EMAILID literally.

"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[@]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words

5
  • In my case error as well, similar if not using quotes at all: termux-contact-list |jq -r '.[] | select(.name=='$1')|.number'. Result: jq Compile error
    – Timo
    Commented Aug 19, 2018 at 7:04
  • 11
    I'm not sure why this comment has been voted up so many times; yes, double quotes allows for variable expansion, but jq doesn't like single quotes: jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?)
    – jdelman
    Commented May 29, 2019 at 19:01
  • 1
    If your input is a regular json, then its values would be double (not single) quoted. What works: Using double quotes for entire arg of jq and escaping (with `\`) any double quotes within, like @asid answered.
    – GuSuku
    Commented Nov 14, 2019 at 17:35
  • It does NOT work. If you put single quotes to delimit the main string, the shell takes $EMAILID literally. Assuming you set $EMAILID="[email protected]", the command cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id" actually runs as cat file.json | jq -r ".resource[] | select([email protected]) | .id" and incur a compile error for jq.
    – axiqia
    Commented Nov 26, 2020 at 12:35
  • This got me on the way to solving my issue. My final working string UPLOADED_ARNS=$(aws devicefarm list-uploads --arn ${PROJECT_ARN} | \ jq ".uploads[] | {arn: .arn, name: .name} | select (.name | contains(\"$FILE_EXTENSION\"))" | \ jq --raw-output '.arn') Commented Sep 30, 2021 at 18:19
13

Jq now have better way to access environment variables, you can use env.EMAILID:

projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
2
  • 1
    Hey DukeLion, it would be nice to precise the min required version to have this working. Your solution didn't work on jq version 1.6.
    – tkrishtop
    Commented Sep 15, 2022 at 9:48
  • I'm using version "jq-1.6" and the above works Commented May 11, 2023 at 14:03
8

Another way to accomplish this is with the jq "--arg" flag. Using the original example:

#!/bin/sh

#this works ***
projectID=$(cat file.json | jq -r '.resource[] | 
select(.username=="[email protected]") | .id')
echo "$projectID"

[email protected]

# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[] 
| select(.username=="$EMAILID") | .id')
echo "$projectID"

See here, which is where I found this solution: https://github.com/stedolan/jq/issues/626

1
  • I had to replace select(.username=="$EMAILID") with select(.username==$EMAILID) to make it work. (I removed the quotes). I'm using jq version 1.6. Commented Jan 17, 2023 at 16:43
4

I know is a bit later to reply, sorry. But that works for me.

export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"

And now I am able to fetch and pass the content of the variable to jq

export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"

In my case I needed to use double quote and quote to access the variable value.

Cheers.

2

All solution above is failed, I got Success from below using one equal '=' not two '==' in select

shell_script | sed '1d'|  jq --arg VAR ${SHELL_VAR} -cC '.[]|select(.branch = $VAR).branch'
1

I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:

RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
0

In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:

$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""
0

it is simple to filter with the variable you set.My case is test on jq-1.6

projectID=$(cat file.json | jq -r ".resource[] | select(.username=="$EMAILID") | .id")"
1
  • Please, edit and try for How to Answer, describe the effect of what you propose and explain why it helps to solve the problem.
    – Yunnosch
    Commented Aug 6 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.