49

I read data from a json file with jq. I wanna append the results into a yaml file, but I dont get it working. I am quite new to shell programming. My goal is to append that "users" to an existing "users"-Array in a yaml file.

This is my json file:

#$DEFAULTS_FILE

{"users":
  [
    {"name":"pi",
      "gecos": "Hypriot Pirate",
      "sudo":"ALL=(ALL) NOPASSWD:ALL",
      "shell": "/bin/bash",
      "groups":"users,docker,video",
      "plain_text_passwd":"pi",
      "lock_passwd":"false",
      "ssh_pwauth":"true",
      "chpasswd": {"expire": false}
    },
    {"name":"admin",
      "gecos": "Hypriot Pirate",
      "sudo":"ALL=(ALL) NOPASSWD:ALL",
      "shell": "/bin/bash",
      "primary-group": "users",
      "groups":"users,docker,adm,dialout,audio,plugdev,netdev,video",
      "ssh-import-id":"None",
      "plain_text_passwd":"pi",
      "lock_passwd":"true",
      "ssh_pwauth":"true",
      "chpasswd": "{expire: false}",
      "ssh-authorized-keys": ["ssh-rsa abcdefg1234567890 YOUR_KEY@YOURHOST.local"]
    }
  ]
  }

I filter it with that:

cat $DEFAULTS_FILE | jq .users

I have no clue how to convert that json into a yaml.

My expected result should be:

users:
  - name:                pi
    gecos:               "Hypriot Pirate"
    sudo:                ALL=(ALL) NOPASSWD:ALL
    shell:               /bin/bash
    groups:              users,docker,video
    plain_text_passwd:   pi
    lock_passwd:         false
    ssh_pwauth:          true
    chpasswd: { expire:  false }
  - name:                admin
    primary-group:       users
    shell:               /bin/bash
    sudo:                ALL=(ALL) NOPASSWD:ALL
    groups:              users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:       None

I tried to use a second tool called yq which is similar to jq and can write yaml files. But I have no positive progress.

EDIT

I know that I can add content to the yaml with that:

yq w -i "my.yml" "users[+]" "some content"

But I dont know how to merge my json into that.

Any help or hint would be nice, thank you in advance...

8
  • 2
    You can use python/perl which have in-built YAML and JSON modules - commandlinefu.com/commands/view/12218/convert-yaml-to-json
    – Inian
    Nov 15, 2018 at 9:11
  • That's exact the opposite direction
    – Jan
    Nov 15, 2018 at 9:18
  • 2
    Bash itself is not a good platform for this. Find an existing tool, or write one in e.g. Python. Depending on your preferred language, it should not be hard to find an existing question on Stack Overflow with suggestions; here's a search for Python
    – tripleee
    Nov 15, 2018 at 10:04
  • Please follow the minimal reproducible example guidelines. In particular, a sample my.yml (i.e. input) would be helpful.
    – peak
    Nov 15, 2018 at 10:17
  • yq r input.json should do the job.
    – AHT
    Nov 15, 2018 at 18:15

10 Answers 10

70

yq a yaml wrapper for jq

With yq 4.18.1+
cat "$DEFAULTS_FILE" | yq -P    # or yq --prettyPrint

See: https://mikefarah.gitbook.io/yq/#notice-for-v4.x-versions-prior-to-4.18.1

With yq version 4.8.0:

cat $DEFAULTS_FILE | yq e -P -

  • e or eval handles file separately. ea or eval-all will merge files first.
  • -P or --prettyPrint YAML output
  • - from STDIN

Note: you can go the other way (yaml to json) too yq e -j file.yaml

With yq version 3.3.2:

cat $DEFAULTS_FILE | yq r -P -

  • r read
  • -P --prettyPrint
  • - from STDIN
7
  • 2
    You're a mad lad! Thanks mate!
    – Robert J
    Sep 29, 2020 at 15:07
  • 1
    This answer needs an edit. Your yq example is an implementation of mikefarah/yq, which is NOT a wrapper over jq. See my answer
    – Inian
    Nov 1, 2020 at 15:25
  • 3
    If using mikefarah/yq V4, you need to use the following command: yq eval '.. style= ""' sample.json or cat sample.json | yq eval '.. style= ""' -. See mikefarah.gitbook.io/yq/usage/convert Dec 25, 2020 at 14:07
  • 1
    Actually, all you need is cat $DEFAULTS_FILE | yq -y
    – Ben Davis
    Jan 3, 2021 at 5:06
  • 4
    With mikefarah/yq version 4, yq eval -P is the correct syntax.
    – ijoseph
    Feb 28, 2021 at 3:58
36
function yaml_validate {
  python -c 'import sys, yaml, json; yaml.safe_load(sys.stdin.read())'
}

function yaml2json {
  python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin.read())))'
}

function yaml2json_pretty {
  python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin.read()), indent=2, sort_keys=False))'
}

function json_validate {
  python -c 'import sys, yaml, json; json.loads(sys.stdin.read())'
}

function json2yaml {
  python -c 'import sys, yaml, json; print(yaml.dump(json.loads(sys.stdin.read())))'
}

More Bash tricks at http://github.com/frgomes/bash-scripts

5
  • Possibly superfluous in light of yq and jq, but very nicely done, and probably has some good applications.
    – abalter
    Oct 2, 2020 at 5:52
  • Ha! As it happens, I ran into errors installing yq just now in a gpc ai notebook. so, I guess these will come in very handy!
    – abalter
    Oct 2, 2020 at 6:00
  • 2
    perhaps the best answer,as it does not make you install yq
    – P....
    Jul 5, 2021 at 17:53
  • awesome-bash-scripts.
    – Akhil
    Dec 28, 2021 at 9:51
  • 1
    yq is bloaty. Thank you.
    – Rondo
    Jan 28, 2022 at 2:24
15

I'm not sure what rules you're using to get to your expected result. It seems like you're randomly applying different rules to how the values are being converted.

As I understand it, scalar values are just output as is (with potential encoding), objects are output as key/value pairs, and array objects are output with a - for every item. The indentation associates what's part of what.

So based on those rules if you're going to use jq:

def yamlify:
    (objects | to_entries[] | (.value | type) as $type |
        if $type == "array" then
            "\(.key):", (.value | yamlify)
        elif $type == "object" then
            "\(.key):", "    \(.value | yamlify)"
        else
            "\(.key):\t\(.value)"
        end
    )
    // (arrays | select(length > 0)[] | [yamlify] |
        "  - \(.[0])", "    \(.[1:][])"
    )
    // .
    ;

Then to use it, add it to your .jq file and use it:

$ jq -r yamlify input.json
users:
  - name:       pi
    gecos:      Hypriot Pirate
    sudo:       ALL=(ALL) NOPASSWD:ALL
    shell:      /bin/bash
    groups:     users,docker,video
    plain_text_passwd:  pi
    lock_passwd:        false
    ssh_pwauth: true
    chpasswd:
        expire: false
  - name:       admin
    gecos:      Hypriot Pirate
    sudo:       ALL=(ALL) NOPASSWD:ALL
    shell:      /bin/bash
    primary-group:      users
    groups:     users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:      None
    plain_text_passwd:  pi
    lock_passwd:        true
    ssh_pwauth: true
    chpasswd:   {expire: false}
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 YOUR_KEY@YOURHOST.local

Here's another variation that aligns the values

def yamlify2:
    (objects | to_entries | (map(.key | length) | max + 2) as $w |
        .[] | (.value | type) as $type |
        if $type == "array" then
            "\(.key):", (.value | yamlify2)
        elif $type == "object" then
            "\(.key):", "    \(.value | yamlify2)"
        else
            "\(.key):\(" " * (.key | $w - length))\(.value)"
        end
    )
    // (arrays | select(length > 0)[] | [yamlify2] |
        "  - \(.[0])", "    \(.[1:][])"
    )
    // .
    ;
$ jq -r yamlify2 input.json
users:
  - name:               pi
    gecos:              Hypriot Pirate
    sudo:               ALL=(ALL) NOPASSWD:ALL
    shell:              /bin/bash
    groups:             users,docker,video
    plain_text_passwd:  pi
    lock_passwd:        false
    ssh_pwauth:         true
    chpasswd:
        expire:  false
  - name:                 admin
    gecos:                Hypriot Pirate
    sudo:                 ALL=(ALL) NOPASSWD:ALL
    shell:                /bin/bash
    primary-group:        users
    groups:               users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:        None
    plain_text_passwd:    pi
    lock_passwd:          true
    ssh_pwauth:           true
    chpasswd:             {expire: false}
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 YOUR_KEY@YOURHOST.local
2
  • 1
    really cool. love it, unfortunately, it breaks for multiline string values..
    – Kaos
    Jun 2, 2020 at 9:57
  • Based on the code of @Jeff Mercado I added some code to support multiline strings and escaping of single quotes. See my answer "Solution in jq (without other tools)".
    – jpseng
    Jun 29, 2021 at 20:01
13

yq eval -P

with mikefarah/yq version 4.0 (released December 2020), installable via most Unix-like OS package managers: via Homebrew for macOS (brew install yq), Debian with apt (apt install yq), Alpine with apk (apk add yq), etc.

See Working with JSON.

To read in json, just pass in a json file instead of yaml, it will just work - as json is a subset of yaml. However, you will probably want to use the Style Operator or --prettyPrint/-P flag to make look more like an idiomatic yaml document.

8

I've used ruby to write my json content into yaml.

As for your example, it can be achieved like this:

cat $DEFAULTS_FILE | jq .users | ruby -ryaml -rjson -e 'puts YAML.dump(JSON.parse(STDIN.read))' > my.yml
2
  • 2
    While I can't stand ruby, my machine already had it installed. I was able to create an alias and pipe it to this alias. Thank you, this has been very helpful!
    – Brandon
    Dec 23, 2021 at 21:05
  • Exact same sentiment here.
    – rookie099
    Mar 30, 2022 at 12:59
7

I suggest using yq with -y option

$ pip3 install yq # requires jq

$ cat in.json | yq -y
users:
  - name: pi
    gecos: Hypriot Pirate
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    groups: users,docker,video
    plain_text_passwd: pi
    lock_passwd: 'false'
    ssh_pwauth: 'true'
    chpasswd:
      expire: false
  - name: admin
    gecos: Hypriot Pirate
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    primary-group: users
    groups: users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id: None
    plain_text_passwd: pi
    lock_passwd: 'true'
    ssh_pwauth: 'true'
    chpasswd: '{expire: false}'
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 YOUR_KEY@YOURHOST.local
5

Another oneliner:

python -c 'import yaml, sys; print(yaml.dump(yaml.load(open(sys.argv[1])), default_flow_style=False))' input.json

(exploiting the fact that valid json is also valid yaml)

And yaml to json:

python -c 'import yaml, json, sys; print(json.dumps(yaml.load(open(sys.argv[1])), indent=2))' input.yaml
1
  • Your yaml-to-json oneliner fails with found character '\t' that cannot start any token; it's the same error message yq reports, which must work similarly. Richard Gomes' one-liner works fine.
    – Bean Taxi
    Mar 29, 2020 at 15:40
2

Solution in jq (without other tools)

Based on the code of @Jeff Mercado in this post I added support for multiline strings and escaping for single quotes.

# purpose: converts Json to Yaml
# remarks:
#   You can use 'yq -y' to convert json to yaml, but ...
#     * this function can be used several times within a single jq program
#     * this function may be faster than using yq
#     * maybe yq is not available in your environment
#
# input: any Json
# output: json converted to yaml
def toYaml:
   def handleMultilineString($level):
      reduce ([match("\n+"; "g")]                       # find groups of '\n'
              | sort_by(-.offset))[] as $match
             (.; .[0:$match.offset + $match.length] +
                 "\n\("    " * $level)" +               # add one extra '\n' for every group of '\n's. Add indention for each new line
                 .[$match.offset + $match.length:]);

   def toYamlString($level):
      if type == "string"
      then handleMultilineString($level)
           | sub("'"; "''"; "g")           # escape single quotes
           | "'\(.)'"                      # wrap in single quotes
      else .
      end;

   def _toYaml($level):
      (objects | to_entries[] |
          if (.value | type) == "array" then
              "\(.key):", (.value | _toYaml($level))
          elif (.value | type) == "object" then
              "\(.key):", "\("    ")\(.value | _toYaml($level))"
          else
              "\(.key): \(.value | toYamlString($level))"
          end
      )
      // (arrays | select(length > 0)[] | [_toYaml($level)] |
          "  - \(.[0])", "\("    ")\(.[1:][])"
      )
      // .;

   _toYaml(1);

Example usage

File 'containsMultilineStrings.json'

{
  "response": {
    "code": 200,
    "message": "greeting\nthat's all folks\n\n\n"
  }
}

jq -r 'toYaml' < containsMultilineStrings.json

response:
    code: 200
    message: 'greeting

    that''s all folks



    '

jq -r 'toYaml' containsMultilineStrings.json | yq (roundtrip)

{
  "response": {
    "code": 200,
    "message": "greeting\nthat's all folks\n\n\n"
  }
}

Test

You can test the correctness of the function toYaml by converting json to yaml and than back to json using yq.

FILE='containsMultilineStrings.json'; diff <(cat "$FILE") <(jq -r 'toYaml' $FILE | yq)

Performance

A quick benchmark shows a reduced runtime of the function toYaml compared to the use of yq. On my computer, I measured:

time for i in {1..100}; do yq -y > /dev/null < containsMultilineStrings.json; done

8.4 sec

time for i in {1..100}; do jq -r 'toYaml' > /dev/null containsMultilineStrings.json; done

3.4 sec

0

Another option is to use gojq. It is a port of jq with support for reading and writing yaml. It can be installed via GitHub releases, homebrew, and zero install. The command for your question would be:

cat test.json | gojq --yaml-output > test.yaml
0

You can use the awesome library yq

# install yq on macOS
brew install yq

Then run the following command

cat file.json | yq -P > file.yml

You can also convert and save a JSON Response into a YAML file, for example:

curl https://raw.githubusercontent.com/typicode/demo/master/db.json | yq -P > example.yml

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.