40

I'm going to preface by saying that "no, find a different way to do it" is an acceptable answer here.

Is there a reliable way to store a short bit of JSON in a bash variable for use in a AWS CLI command running from the same script?

I'll be running a job from Jenkins that's updating an AWS Route53 record, which requires UPSERTing a JSON file with the change in records. Because it's running from Jenkins, there's no local storage where I can keep this file, and I'd really like to avoid needing to do a git checkout every time this project will run (which will be once an hour).

Ideally, storing the data in a variable ($foo) and calling it as part of the change-resource-record-sets command would be most convenient given the Jenkins setup, but I'm unfamiliar with exactly how to quote/store JSON inside bash - can it be done safely?

The specific JSON in this case is the following;

{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"$bar","EvaluateTargetHealth":false}}}]}

As an added complication the DNSName value - $bar - needs to be expanded.

2 Answers 2

79

You could use a here-doc:

foo=$(cat <<EOF
{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"$bar","EvaluateTargetHealth":false}}}]}
EOF
)

By leaving EOF in the first line unquoted, the contents of the here-doc will be subject to parameter expansion, so your $bar expands to whatever you put in there.

If you can have linebreaks in your JSON, you can make it a little more readable:

foo=$(cat <<EOF
{
  "Comment": "Update DNSName.",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "alex.",
        "Type": "A",
        "AliasTarget": {
          "HostedZoneId": "######",
          "DNSName": "$bar",
          "EvaluateTargetHealth": false
        }
      }
    }
  ]
}
EOF
)

or even (first indent on each line must be a tab)

foo=$(cat <<-EOF
    {
      "Comment": "Update DNSName.",
      "Changes": [
        {
          "Action": "UPSERT",
          "ResourceRecordSet": {
            "Name": "alex.",
            "Type": "A",
            "AliasTarget": {
              "HostedZoneId": "######",
              "DNSName": "$bar",
              "EvaluateTargetHealth": false
            }
          }
        }
      ]
    }
    EOF
)

and to show how that is stored, including quoting (assuming that bar=baz):

$ declare -p foo
declare -- foo="{
  \"Comment\": \"Update DNSName.\",
  \"Changes\": [
    {
      \"Action\": \"UPSERT\",
      \"ResourceRecordSet\": {
        \"Name\": \"alex.\",
        \"Type\": \"A\",
        \"AliasTarget\": {
          \"HostedZoneId\": \"######\",
          \"DNSName\": \"baz\",
          \"EvaluateTargetHealth\": false
        }
      }
    }
  ]
}"

Because this expands some shell metacharacters, you could run into trouble if your JSON contains something like `, so alternatively, you could assign directly, but be careful about quoting around $bar:

foo='{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"'"$bar"'","EvaluateTargetHealth":false}}}]}'

Notice the quoting for $bar: it's

"'"$bar"'"
│││    │││
│││    ││└ literal double quote
│││    │└ opening syntactical single quote
│││    └ closing syntactical double quote
││└ opening syntactical double quote
│└ closing syntactical single quote
└ literal double quote
4
  • 2
    This worked perfectly, and especially thanks for the quoting breakdown for bar, that solved a future problem!
    – Alex
    Apr 12, 2017 at 15:36
  • Not only is that the solution I wanted, but it is the exact use-case. Thanks Sep 5, 2018 at 16:56
  • closing syntactical single quote and opening syntactical single quote the other way round?
    – Timo
    May 13, 2021 at 17:38
  • 1
    @Timo No, they're labeled correctly: the inner "$bar" is outside of the single-quoted string, so the first ' closes a single quoted string, and the second ' opens a new one. May 13, 2021 at 17:52
18

It can be stored safely; generating it is a different matter, since the contents of $bar may need to be encoded. Let a tool like jq handle creating the JSON.

var=$(jq -n --arg b "$bar" '{
  Comment: "Update DNSName.",
  Changes: [
    {
      Action: "UPSERT",
      ResourceRecordSet: {
        Name: "alex.",
        Type: "A",
        AliasTarget: {
          HostedZoneId: "######",
          DNSName: $b,
          EvaluateTargetHealth: false
        }
      }
    }
  ]
}')
6
  • jq is absolutely an option as we use it elsewhere, but I'm wondering about the lack of quotes - does jq handle putting quotes around everything or is that just something you left out in your example? AWS seems to demand quoted keypairs.
    – Alex
    Apr 12, 2017 at 15:25
  • 2
    @Alex jq allows a key in a filter to be unquoted if it is a simply alphanumeric key; the generated JSON will have quotes.
    – chepner
    Apr 12, 2017 at 15:45
  • 1
    This is the solution, because the jq checks the JSON so is possible catch eventual errors, and be sure that the generated JSON is always correct. (Aka: using right tool for the problem) :)
    – clt60
    Apr 12, 2017 at 15:53
  • 1
    n from man jq: Don´t read any input at all! Instead, the filter is run once using null as the input. This is useful when using jq as a simple calculator or to construct JSON data from scratch. --- Sounds to me like not appending further data, just one time json creation.
    – Timo
    May 13, 2021 at 17:53
  • 1
    You didn't quote $var, so it is subject to word splitting. Only the part of $var preceding its first space is used as the argument to -d; the next bit is treated as the host name, not localhost.
    – chepner
    May 13, 2021 at 18:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.