229

I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).

#!/bin/sh

#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail@hotmail.com") | .id')
echo "$projectID"

EMAILID=myemail@hotmail.com

#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
2
  • 8
    A related issue: passing bash variable to jq filter has the slightly different syntax jq -r --arg var "$var" '.[$var]' stackoverflow.com/questions/34745451/…
    – enharmonic
    Feb 19, 2020 at 18:11
  • EMAILIDis between single quotes and does not get expanded. Write it as '"$EMAILID"' to take it out of the single quotes. Mar 24 at 8:53

10 Answers 10

350

Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):

   projectID=$(jq -r --arg EMAILID "$EMAILID" '
        .resource[]
        | select(.username==$EMAILID) 
        | .id' file.json)

Postscript

For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:

EMAILID=foo@bar.com  # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'

The output is a JSON string:

"foo@bar.com"

shell arrays

Unfortunately, shell arrays are a different kettle of fish. Here are two SO resources regarding the ingestion of such arrays:

JQ - create JSON array using bash array with space

Convert bash array to json array and insert to file using jq

9
  • 5
    This is the only 100%-safe answer; it lets jq properly create the filter using the value, rather than using bash to create a string that jq interprets as a filter. (Consider what happens if the value of EMAILID contains a ).)
    – chepner
    Oct 16, 2016 at 2:35
  • 2
    My use case, it works! function n2 { termux-contact-list |jq -r --arg v1 "$1" '.[] | select(.name==$v1)|.number' } Calling: n2 Name1
    – Timo
    Aug 19, 2018 at 7:12
  • 6
    fyi, jq's env function is changed between jq 1.4 and 1.5+, so references to env.EMAILID (in this answer's example) do not work in jq 1.4, thus recommending using the --arg EMAILID "$EMAILID" construct if you need to use jq 1.4 . Hope this helps; just took me a day+ to figure this out for myself ;)
    – m0j0hn
    Oct 4, 2018 at 20:15
  • 1
    In my case I had to use parenthesis around the variable name, and since it was inside a select I had to escape it too, like this jq --arg var "${SHELL_VAR}" '.UserPools[] | select(.Name == "\($var)-some-string")'
    – Federico
    May 23, 2019 at 2:24
  • 5
    Arguments passed using --arg will be passed as strings. If you want to pass data of a different JSON type, use --argjson, which will parse the string as JSON, e.g., --argjson myObj '{"a": [1, 2]}' Feb 17, 2020 at 16:59
51

I resolved this issue by escaping the inner double quotes

projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
1
  • 1
    I am using this because --arg doesn't play nice with -c
    – dimisjim
    Apr 30, 2020 at 12:56
34

Little unrelated but I will still put it here, For other practical purposes shell variables can be used as -

value=10
jq  '."key" = "'"$value"'"' file.json
27

Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:

.items[] | select(.name=="string")

in bash you could do

EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')

essentially escaping the quotes and passing it on to jq

0
17

It's a quote issue, you need :

projectID=$(
  cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)

If you put single quotes to delimit the main string, the shell takes $EMAILID literally.

"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[@]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words

5
  • In my case error as well, similar if not using quotes at all: termux-contact-list |jq -r '.[] | select(.name=='$1')|.number'. Result: jq Compile error
    – Timo
    Aug 19, 2018 at 7:04
  • 11
    I'm not sure why this comment has been voted up so many times; yes, double quotes allows for variable expansion, but jq doesn't like single quotes: jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?)
    – jdelman
    May 29, 2019 at 19:01
  • 1
    If your input is a regular json, then its values would be double (not single) quoted. What works: Using double quotes for entire arg of jq and escaping (with `\`) any double quotes within, like @asid answered.
    – GuSuku
    Nov 14, 2019 at 17:35
  • It does NOT work. If you put single quotes to delimit the main string, the shell takes $EMAILID literally. Assuming you set $EMAILID="myemail@hotmail.com", the command cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id" actually runs as cat file.json | jq -r ".resource[] | select(.username==myemail@hotmail.com) | .id" and incur a compile error for jq.
    – axiqia
    Nov 26, 2020 at 12:35
  • This got me on the way to solving my issue. My final working string UPLOADED_ARNS=$(aws devicefarm list-uploads --arn ${PROJECT_ARN} | \ jq ".uploads[] | {arn: .arn, name: .name} | select (.name | contains(\"$FILE_EXTENSION\"))" | \ jq --raw-output '.arn') Sep 30, 2021 at 18:19
11

Jq now have better way to access environment variables, you can use env.EMAILID:

projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
1
  • 1
    Hey DukeLion, it would be nice to precise the min required version to have this working. Your solution didn't work on jq version 1.6.
    – tkrishtop
    Sep 15, 2022 at 9:48
7

Another way to accomplish this is with the jq "--arg" flag. Using the original example:

#!/bin/sh

#this works ***
projectID=$(cat file.json | jq -r '.resource[] | 
select(.username=="myemail@hotmail.com") | .id')
echo "$projectID"

EMAILID=myemail@hotmail.com

# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[] 
| select(.username=="$EMAILID") | .id')
echo "$projectID"

See here, which is where I found this solution: https://github.com/stedolan/jq/issues/626

1
  • I had to replace select(.username=="$EMAILID") with select(.username==$EMAILID) to make it work. (I removed the quotes). I'm using jq version 1.6. Jan 17 at 16:43
2

I know is a bit later to reply, sorry. But that works for me.

export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"

And now I am able to fetch and pass the content of the variable to jq

export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"

In my case I needed to use double quote and quote to access the variable value.

Cheers.

1

I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:

RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
0

In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:

$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.