114

I am trying to work out how to make bash (force?) expand variables in a string (which was loaded from a file).

I have a file called "something.txt" with the contents:

hello $FOO world

I then run

export FOO=42
echo $(cat something.txt)

this returns:

   hello $FOO world

It didn't expand $FOO even though the variable was set. I can't eval or source the file - as it will try and execute it (it isn't executable as it is - I just want the string with the variables interpolated).

Any ideas?

5
  • 3
    Be aware of the security implications of eval. May 21, 2012 at 13:08
  • Did you mean the comment to be for the answer below? May 22, 2012 at 3:41
  • I meant the comment for you. More than one answer proposes the use of eval and it's important to be aware of the implications. May 22, 2012 at 3:50
  • @DennisWilliamson gotcha - I am a bit late replying but good tip. And of course in the intervening years we have had things like "shell shock" so your comment has stood the test of time! tips hat Mar 17, 2017 at 4:04
  • Does this answer your question? Bash expand variable in a variable
    – LoganMzz
    Jan 17, 2020 at 10:43

14 Answers 14

145

I stumbled on what I think is THE answer to this question: the envsubst command:

echo "hello \$FOO world" > source.txt
export FOO=42
envsubst < source.txt

This outputs: hello 42 world

If you would like to continue work on the data in a file destination.txt, push this back to a file like this:

envsubst < source.txt > destination.txt

In case it's not already available in your distro, it's in the GNU package gettext.

@Rockallite

  • I wrote a little wrapper script to take care of the '$' problem.

(BTW, there is a "feature" of envsubst, explained at https://unix.stackexchange.com/a/294400/7088 for expanding only some of the variables in the input, but I agree that escaping the exceptions is much more convenient.)

Here's my script:

#! /bin/bash
      ## -*-Shell-Script-*-
CmdName=${0##*/}
Usage="usage: $CmdName runs envsubst, but allows '\$' to  keep variables from
    being expanded.
  With option   -sl   '\$' keeps the back-slash.
  Default is to replace  '\$' with '$'
"

if [[ $1 = -h ]]  ;then echo -e >&2  "$Usage" ; exit 1 ;fi
if [[ $1 = -sl ]] ;then  sl='\'  ; shift ;fi

sed 's/\\\$/\${EnVsUbDolR}/g' |  EnVsUbDolR=$sl\$  envsubst  "$@"
5
  • 6
    +1. This is the only answer that's guaranteed to not do command substitutions, quote removal, argument processing, etc. Aug 13, 2015 at 20:42
  • 6
    It just works for fully exported shell vars though (in bash). e.g. S='$a $b'; a=set; export b=exported; echo $S |envsubst; yields: " exported"
    – gaoithe
    Aug 19, 2015 at 13:46
  • 1
    thanks - I think after all these years, I should accept this answer. Mar 17, 2017 at 4:05
  • 1
    However, it doesn't handle dollar sign ($) escaping, e.g. echo '\$USER' | envsubst will output the current username with a prefixing backward slash, not $USER.
    – Rockallite
    Apr 8, 2017 at 9:54
  • 3
    seems to get this on a mac with homebrew needs brew link --force gettext Dec 22, 2017 at 0:03
29

you can try

echo $(eval echo $(cat something.txt))
5
  • 12
    Use echo -e "$(eval "echo -e \"`<something.txt`\"")" if you need new lines kept.
    – pevik
    Sep 13, 2013 at 10:20
  • Works like a charm
    – kyb
    Apr 6, 2018 at 8:08
  • 1
    But only if your template.txt is text. This operation is dangerous because it executes(evalutes) commands if they are.
    – kyb
    Apr 6, 2018 at 9:23
  • 3
    but this removed double-quote May 16, 2018 at 13:05
  • Blame it on the subshell.
    – ingyhere
    Jun 4, 2019 at 21:39
29

Many of the answers using eval and echo kind of work, but break on various things, such as multiple lines, attempting to escaping shell meta-characters, escapes inside the template not intended to be expanded by bash, etc.

I had the same issue, and wrote this shell function, which as far as I can tell, handles everything correctly. This will still strip only trailing newlines from the template, because of bash's command substitution rules, but I've never found that to be an issue as long as everything else remains intact.

apply_shell_expansion() {
    declare file="$1"
    declare data=$(< "$file")
    declare delimiter="__apply_shell_expansion_delimiter__"
    declare command="cat <<$delimiter"$'\n'"$data"$'\n'"$delimiter"
    eval "$command"
}

For example, you can use it like this with a parameters.cfg which is really a shell script that just sets variables, and a template.txt which is a template that uses those variables:

. parameters.cfg
printf "%s\n" "$(apply_shell_expansion template.txt)" > result.txt

In practice, I use this as a sort of lightweight template system.

3
  • 5
    This is one of the best tricks I found so far :). Based on your answer it's so easy to built a function that expand a variable with a string containing references to other variables -- just replace $data with $1 in your function and here we go! I believe this is the best answer to the original question, BTW.
    – galaxy
    Jun 17, 2014 at 11:10
  • Even this has usual eval pitfalls. Try adding ; $(eval date) at the end of any line inside input file and it will run date command.
    – anubhava
    Jun 30, 2016 at 16:07
  • 3
    @anubhava I'm not sure what you mean; that is actually the desired expansion behavior in this case. In other words, yes, you are supposed to be able to execute arbitrary shell code with this.
    – wjl
    Jul 3, 2016 at 18:59
8

You don't want to print each line, you want to evaluate it so that Bash can perform variable substitutions.

FOO=42
while read; do
    eval echo "$REPLY"
done < something.txt

See help eval or the Bash manual for more information.

1
  • 2
    eval is dangerous; you get not only $varname expanded (as you would with envsubst), but $(rm -rf ~) as well. Sep 21, 2018 at 1:45
7

Another approach (which seems icky, but I am putting it here anyway):

Write the contents of something.txt to a temp file, with an echo statement wrapped around it:

something=$(cat something.txt)

echo "echo \"" > temp.out
echo "$something" >> temp.out
echo "\"" >> temp.out

then source it back in to a variable:

RESULT=$(source temp.out)

and the $RESULT will have it all expanded. But it seems so wrong !


Single line solution that doesn't need temporary file :

RESULT=$(source <(echo "echo \"$(cat something.txt)\""))
#or
RESULT=$(source <(echo "echo \"$(<something.txt)\""))
1
  • 1
    icky but it is most straight-forward, simple, and works.
    – kyb
    Apr 6, 2018 at 10:31
4

If you only want the variable references to be expanded (an objective that I had for myself) you could do the below.

contents="$(cat something.txt)"
echo $(eval echo \"$contents\")

(The escaped quotes around $contents is key here)

3
  • I have tried this one, however the $() removes line endings in my case which breaks the resulting string
    – khusseini
    May 15, 2018 at 14:58
  • I changed the eval line to eval "echo \"$$contents\"" and it preserved the whitespace
    – khusseini
    May 15, 2018 at 15:24
  • @khusseini I think you mean eval "echo \"$contents\"" with a single dollar sign? $$contents is the process ID of the script followed by the word contents. May 26, 2022 at 2:15
3
  1. If something.txt has only one line, a bash method, (a shorter version of Michael Neale's "icky" answer), using process & command substitution:

    FOO=42 . <(echo -e echo $(<something.txt))
    

    Output:

    hello 42 world
    

    Note that export isn't needed.

  2. If something.txt has one or more lines, a GNU sed evaluate method:

    FOO=42 sed 's/"/\\\"/g;s/.*/echo "&"/e' something.txt
    
1
  • 1
    I found the sed evaluate to best fit my purpose. I needed to produce scripts from templates, which would have values filled from variables exported in another config file. It handles properly escaping for command expansion e.g. put \$(date) in the template to get $(date) in the output. Thanks!
    – gadamiak
    Jan 4, 2019 at 11:39
1

Following solution:

  • allows replacing of variables which are defined

  • leaves unchanged variables placeholders which are not defined. This is especially useful during automated deployments.

  • supports replacement of variables in following formats:

    ${var_NAME}

    $var_NAME

  • reports which variables are not defined in environment and returns error code for such cases



    TARGET_FILE=someFile.txt;
    ERR_CNT=0;

    for VARNAME in $(grep -P -o -e '\$[\{]?(\w+)*[\}]?' ${TARGET_FILE} | sort -u); do     
      VAR_VALUE=${!VARNAME};
      VARNAME2=$(echo $VARNAME| sed -e 's|^\${||g' -e 's|}$||g' -e 's|^\$||g' );
      VAR_VALUE2=${!VARNAME2};

      if [ "xxx" = "xxx$VAR_VALUE2" ]; then
         echo "$VARNAME is undefined ";
         ERR_CNT=$((ERR_CNT+1));
      else
         echo "replacing $VARNAME with $VAR_VALUE2" ;
         sed -i "s|$VARNAME|$VAR_VALUE2|g" ${TARGET_FILE}; 
      fi      
    done

    if [ ${ERR_CNT} -gt 0 ]; then
        echo "Found $ERR_CNT undefined environment variables";
        exit 1 
    fi
0
1
foo=45
file=something.txt       # in a file is written: Hello $foo world!
eval echo $(cat $file)
1
  • 1
    Thank you for this code snippet, which might provide some limited, immediate help. A proper explanation would greatly improve its long-term value by showing why this is a good solution to the problem and would make it more useful to future readers with other, similar questions. Please edit your answer to add some explanation, including the assumptions you’ve made.
    – Capricorn
    Sep 11, 2018 at 16:52
0
$ eval echo $(cat something.txt)
hello 42 world
$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
Copyright (C) 2007 Free Software Foundation, Inc.
0
catexp () {
    local function_name="${FUNCNAME[0]}"
    local uuid="$(uuidgen)"
    if [[ $# = 0 && -t 0 ]]; then
        echo "$function_name: Missing argument and stdin." >&2
        return 1
    fi
    . <(
        printf -- '%s\n' "cat << ${uuid}"
        cat "$@" 2> >(perl -spe 's/^cat: /$function_name: /' -- -function_name="$function_name" >&2)
        exit_code=$?
        printf -- '\n%s\n' "${uuid}"
        printf -- '%s\n' "(exit ${exit_code})"
    ) | perl -0pe 's/\n$//'
    pipestatus
    return $?
}

pipestatus () {
    set "${PIPESTATUS[@]}"
    local i
    for (( i=1; i<$#; i++ )); do
        if [[ ${!i} -ge 1 ]]; then break; fi
    done
    return ${!i}
}

echo "hello \$FOO world" > source.txt
FOO=42 # Also without export
catexp source.txt

Or:

catexp < source.txt

Or:

cat source.txt | catexp

Also:

catexp file1 file2 file3 ...
-1

envsubst is a great solution (see LenW's answer) if the content you're substituting is of "reasonable" length.

In my case, I needed to substitute in a file's content to replace the variable name. envsubst requires that the content be exported as environment variables and bash has a problem when exporting environment variables that are more than a megabyte or so.

awk solution

Using cuonglm's solution from a different question:

needle="doc1_base64" # The "variable name" in the file. (A $ is not needed.)
needle_file="doc1_base64.txt" # Will be substituted for the needle 
haystack=$requestfile1 # File containing the needle
out=$requestfile2
awk "BEGIN{getline l < \"${needle_file}\"}/${needle}/{gsub(\"${needle}\",l)}1" $haystack > $out

This solution works for even large files.

-1
expenv () {
    LF=$'\n'
    echo "cat <<END_OF_TEXT${LF}$(< "$1")${LF}END_OF_TEXT" | bash
    return $?
}

expenv "file name"
1
-3

The following works: bash -c "echo \"$(cat something.txt)"\"

1
  • This is not just inefficient, but also extremely dangerous; it doesn't just run safe expansions like $foo, but unsafe ones like $(rm -rf ~). Sep 21, 2018 at 1:45

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.