5

I'd like to use Sed to expand variables inside a file.

Suppose I exported a variable VARIABLE=something, and have a "test" file with the following:

I'd like to expand this: "${VARIABLE}"

I've been trying commands like the following, but to no avail:

cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile

The result is the "outputfile" with the variable still not expanded:

I'd like to expand this: "${VARIABLE}"

Still, running eval "echo '${VARIABLE}' in bash console results in the value "something" being echoed. Also, I tested and that pattern is trully being matched.

The desired output would be

I'd like to expand this: "something"

Can anyone shed a light on this?

1
  • 1
    This doesn't work because the backticked command substitution is run only once by the shell, before sed is ever invoked. It evaluates to \1, and sed proceeds with 's/..../\1/'.
    – Cascabel
    Oct 22, 2009 at 19:53

2 Answers 2

13

Consider your trial version:

cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile

The reason this doesn't work is because it requires prescience on the part of the shell. The sed script is generated before any pattern is matched by sed, so the shell cannot do that job for you.

I've done this a couple of ways in the past. Normally, I've had a list of known variables and their values, and I've done the substitution from that list:

for var in PATH VARIABLE USERNAME
do
    echo 's%${'"$var"'}%'$(eval echo "\$$var")'%g'
done > sed.script

cat test | sed -f sed.script > outputfile

If you want to map variables arbitrarily, then you either need to deal with the whole environment (instead of the fixed list of variable names, use the output from env, appropriately edited), or use Perl or Python instead.

Note that if the value of an environment variable contains a slash in your version, you'd run into problems using the slash as the field separator in the s/// notation. I used the '%' since relatively few environment variables use that - but there are some found on some machines that do contain '%' characters and so a complete solution is trickier. You also need to worry about backslashes in the value. You probably have to use something like '$(eval echo "\$$var" | sed 's/[\%]/\\&/g')' to escape the backslashes and percent symbols in the value of the environment variable. Final wrinkle: some versions of sed have (or had) a limited capacity for the script size - older versions of HP-UX had a limit of about 100. I'm not sure whether that is still an issue, but it was as recently as 5 years ago.

The simple-minded adaptation of the original script reads:

env |
sed 's/=.*//' |
while read var
do
    echo 's%${'"$var"'}%'$(eval echo "\$$var" | sed 's/[\%]/\\&/g')'%g'
done > sed.script

cat test | sed -f sed.script > outputfile

However, a better solution uses the fact that you already have the values in the output from env, so we can write:

env |
sed 's/[\%]/\\&/g;s/\([^=]*\)=\(.*\)/s%${\1}%\2%/' > sed.script
cat test | sed -f sed.script > outputfile

This is altogether safer because the shell never evaluates anything that should not be evaluated - you have to be so careful with shell metacharacters in variable values. This version can only possibly run into any trouble if some output from env is malformed, I think.

Beware - writing sed scripts with sed is an esoteric occupation, but one that illustrates the power of good tools.

All these examples are remiss in not cleaning up the temporary file(s).

1
  • some ten years later, this answer is still nice. I especially like the last version of your script
    – swe
    Aug 10, 2018 at 6:45
6

Maybe you can get by without using sed:

$ echo $VARIABLE
something
$ cat test
I'd like to expand this: ${VARIABLE}
$ eval "echo \"`cat test`\"" > outputfile
$ cat outputfile
I'd like to expand this: something

Let shell variable interpolation do the work.

3
  • This almost solved it. I failed to add that the variable in the files I'll work are typed within double-quotes, something along the lines of ...I'd like to expand this: "${VARIABLE}"... In that case, this sequence will produce an output where the variable is expanded, but the double-quotes are gone.
    – jmissao
    Oct 22, 2009 at 22:35
  • 2
    That will fail horribly if the test file contains anything more dangerous - like: $(rm -fr $HOME) (or the back-quoted version of that). Oct 22, 2009 at 22:39
  • 2
    Indeed - not at all suitable for tainted input. Oct 22, 2009 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.