0

im trying to create yaml files from a template, using my variables. My yaml template look like this

number: {{NUMBER}}
  name: {{NAME}}
  region: {{REGION}}
  storenum: {{STORENUM}}
  clients: {{CLIENTS}}
  tags: {{TAGS}}


storename: {{STORENAME}}
employee: {{EMPLOYEE}}
products: {{PRODUCTS}}

But my variables are in a CSV file the structure is the variables.

Number - Name - Region - Storenum  
StoreX - StoreX - New York - 30  

I now have a little script, to create from a template with the variable parameters and the template like this script.sh template.yml -f variables.txt. And my result look like this

number: 37579922
  name: Store1
  region: New York
  storenum: 32
  clients: 100
  tags: stores


storename: Store newyork
employee: 10
products: 200

But i can only do one by one. Is there any way to read the CSV parameters and send to the program and generate for example Template1,Template2,.. from the CSV parameters? Any help

#!/bin/bash
readonly PROGNAME=$(basename $0)

config_file="<none>"
print_only="false"
silent="false"

usage="${PROGNAME} [-h] [-d] [-f] [-s] -- 

where:
    -h, --help
        Show this help text
    -p, --print
        Don't do anything, just print the result of the variable expansion(s)
    -f, --file
        Specify a file to read variables from
    -s, --silent
        Don't print warning messages (for example if no variables are found)

examples:
    VAR1=Something VAR2=1.2.3 ${PROGNAME} test.txt 
    ${PROGNAME} test.txt -f my-variables.txt
    ${PROGNAME} test.txt -f my-variables.txt > new-test.txt"

if [ $# -eq 0 ]; then
  echo "$usage"
  exit 1    
fi

if [[ ! -f "${1}" ]]; then
    echo "You need to specify a template file" >&2
    echo "$usage"
    exit 1
fi

template="${1}"

if [ "$#" -ne 0 ]; then
    while [ "$#" -gt 0 ]
    do
        case "$1" in
        -h|--help)
            echo "$usage"
            exit 0
            ;;        
        -p|--print)
            print_only="true"
            ;;
        -f|--file)
            config_file="$2"
            ;;
        -s|--silent)
            silent="true"
            ;;
        --)
            break
            ;;
        -*)
            echo "Invalid option '$1'. Use --help to see the valid options" >&2
            exit 1
            ;;
        # an option argument, continue
        *)  ;;
        esac
        shift
    done
fi

vars=$(grep -oE '\{\{[A-Za-z0-9_]+\}\}' "${template}" | sort | uniq | sed -e 's/^{{//' -e 's/}}$//')

if [[ -z "$vars" ]]; then
    if [ "$silent" == "false" ]; then
        echo "Warning: No variable was found in ${template}, syntax is {{VAR}}" >&2
    fi
fi

# Load variables from file if needed
if [ "${config_file}" != "<none>" ]; then
    if [[ ! -f "${config_file}" ]]; then
      echo "The file ${config_file} does not exists" >&2
      echo "$usage"      
      exit 1
    fi

    source "${config_file}"
fi    

var_value() {
    eval echo \$$1
}

replaces=""

# Reads default values defined as {{VAR=value}} and delete those lines
# There are evaluated, so you can do {{PATH=$HOME}} or {{PATH=`pwd`}}
# You can even reference variables defined in the template before
defaults=$(grep -oE '^\{\{[A-Za-z0-9_]+=.+\}\}' "${template}" | sed -e 's/^{{//' -e 's/}}$//')

for default in $defaults; do
    var=$(echo "$default" | grep -oE "^[A-Za-z0-9_]+")
    current=`var_value $var`

    # Replace only if var is not set
    if [[ -z "$current" ]]; then
        eval $default
    fi

    # remove define line
    replaces="-e '/^{{$var=/d' $replaces"
    vars="$vars
$current"
done

vars=$(echo $vars | sort | uniq)

if [[ "$print_only" == "true" ]]; then
    for var in $vars; do
        value=`var_value $var`
        echo "$var = $value"
    done
    exit 0
fi

# Replace all {{VAR}} by $VAR value
for var in $vars; do
    value=$(var_value $var | sed -e "s;\&;\\\&;g" -e "s;\ ;\\\ ;g") # '&' and <space> is escaped 
    if [[ -z "$value" ]]; then
        if [ $silent == "false" ]; then
            echo "Warning: $var is not defined and no default is set, replacing by empty" >&2
        fi
    fi

    # Escape slashes
    value=$(echo "$value" | sed 's/\//\\\//g');
    replaces="-e 's/{{$var}}/${value}/g' $replaces"    
done

escaped_template_path=$(echo $template | sed 's/ /\\ /g')
eval sed $replaces "$escaped_template_path"
4
  • please update the question with a few (complete) sample lines from the CSV file; will the values in the CSV file always be in the same position or do we need to parse the header row to determine which value is in which field?; also update the question with the expected output (matching the sample lines from the CSV file); lastly, consider reducing the code block to just the code needed to parse the CSV file and generate the output (ie, we don't need to see the usage nor all of the command line option processing)
    – markp-fuso
    Oct 2, 2021 at 17:58
  • I haven't tried to decipher all of the code but the references to defaults leads me to question ... what do we do if the CSV file does not have a value for a template field? how would we know this scenario occurs and what are the 'default' values we should be using (ie, show the file and/or structure that contains the 'default' values)
    – markp-fuso
    Oct 2, 2021 at 17:59
  • hello @markp-fuso, all the fields in the CSV are the same with the same structure. Number - Name - Region - Storenum - ..
    – Jo0l
    Oct 2, 2021 at 18:35
  • I wrote a little tool in powershell to expand a template repetitively using the data from a CSV file to replace the variables in the template. In powershell, it is fairly simple to setp through the fields of a CSV record, and create a powershell variable with the right name and the right value. You can see my tool here. I don't know how to do it in bash. Feb 13, 2022 at 17:05

2 Answers 2

0

But i can only do one by one. Is there any way to read the CSV parameters and send to the program and generate for example Template1,Template2,.. from the CSV parameters? Any help

I can't come up with a complete solution but I have a solution that can create multiple files. Just adopt it to your script.


The file.csv

Number,Name,Region,Storenum,Clients,Tags,Storename,Employee,Products
88899223,Store1,New York,30,100,stores,Store newyork,10,200
37579922,Store2,Chicago,30,1000,stores,Store Chicago,10,200
77777777,Store3,New Orleans,309,100,stores,Store New Orleans,10,200
55555555,Store4,New Jersey,50,100,stores,Store Jersey,10,200
44444444,Store5,Connecticut,90,100,stores,Store Connecticut,10,200
33333333,Store6,Michigan,,900,stores,Store Michigan,10,200
22222222,Store7,Texas,30,200,stores,,10,200

The Template.yaml

number: {{NUMBER}}
  name: {{NAME}}
  region: {{REGION}}
  storenum: {{STORENUM}}
  clients: {{CLIENTS}}
  tags: {{TAGS}}


storename: {{STORENAME}}
employee: {{EMPLOYEE}}
products: {{PRODUCTS}}

#!/usr/bin/env bash

n=1
skip=0
start=-1

while IFS=, read -r number name region storenum clients tags storename employee products; do
  if ((start++ >= skip)); then
    export NUMBER="${number:-none}" NAME="${name:-none}" REGION="${region:-none}" \
      STORENUM="${storenum:-none}" CLIENTS="${clients:-none}" TAGS="${tags:-none}" \
      STORENAME="${storename:-none}" EMPLOYEE="${employee:-none}" PRODUCTS="${products:-none}"
   sed 's/{{/$/;s/}}//g' Template.yaml |
   envsubst '$NUMBER $NAME $REGION $STORENUM $CLIENTS $TAGS $STORENAME $EMPLOYEE $PRODUCTS' > "Template$n"
  ((n++))
  fi
done < file.csv

Checking the created files by running:

tail -n+1 Template[0-9]*

The output is:

==> Template1 <==
number: 88899223
  name: Store1
  region: New York
  storenum: 30
  clients: 100
  tags: stores


storename: Store newyork
employee: 10
products: 200


==> Template2 <==
number: 37579922
  name: Store2
  region: Chicago
  storenum: 30
  clients: 1000
  tags: stores


storename: Store Chicago
employee: 10
products: 200


==> Template3 <==
number: 77777777
  name: Store3
  region: New Orleans
  storenum: 309
  clients: 100
  tags: stores


storename: Store New Orleans
employee: 10
products: 200


==> Template4 <==
number: 55555555
  name: Store4
  region: New Jersey
  storenum: 50
  clients: 100
  tags: stores


storename: Store Jersey
employee: 10
products: 200


==> Template5 <==
number: 44444444
  name: Store5
  region: Connecticut
  storenum: 90
  clients: 100
  tags: stores


storename: Store Connecticut
employee: 10
products: 200


==> Template6 <==
number: 33333333
  name: Store6
  region: Michigan
  storenum: none
  clients: 900
  tags: stores


storename: Store Michigan
employee: 10
products: 200


==> Template7 <==
number: 22222222
  name: Store7
  region: Texas
  storenum: 30
  clients: 200
  tags: stores


storename: none
employee: 10
products: 200


==> Template8 <==
number: 22222222
  name: Store7
  region: Texas
  storenum: 30
  clients: 200
  tags: stores


storename: none
employee: 10
products: 200

  • The above script requires/needs envsubst and sed since export is a builtin from the bash shell.

  • If there are no value in the field then the default is none.

  • Caveat: The shell is inherently slow for big file/data size.

0

This never got an accepted answer. I'm offering a second partial solution.

Start with this:

Import-Csv MyFile.csv | ConvertTo-Json | OutFile myfile.json

The outfile at the end is just a dummy that stores the pipeline result in a file. your can do anything you want with the pipeline output.

Now there are two things wrong here: It introduces some quote marks you don't want. And it generates JSON not YAML.

There is a module in the powershell Gallery named Powershell-Yaml. It includes a tool like ConvertTo-Json. you can import the module and fix it up to do what you want.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.