46

I'm working on a bash-script that has to prepare an E-Mail for being sent to a user.

It aggregates some data, which ends up being multiple lines of stuff. For the example stored in $DATA.

Now, after a bit of stfw I found a few things like sed -ei "s/_data_/${DATA}/g" mail.tpl and also sed replace with variable with multiple lines. None of them work.

Now the question is, how do I get sed to replace something with multiple lines of text?

(Alternatives to sed are also welcome!)

5
  • 3
    @ZsoltBotykai You dont say? :D I even mentioned it in my question...
    – Cobra_Fast
    Apr 11, 2012 at 14:07
  • As an aside, don't use uppercase for your private shell variables.
    – tripleee
    Mar 11, 2018 at 12:23
  • @tripleee why not?
    – Cobra_Fast
    Mar 11, 2018 at 15:44
  • 1
    Because that's how you avoid inadvertent clashes with reserved variables, which are uppercase. There's no consensus on what exactly this means but your script's private variables are definitely not "system variables" as intended in this clause in POSIX. See stackoverflow.com/q/673055/874188
    – tripleee
    Mar 11, 2018 at 15:54
  • @Cobra_Fast any chance you could consider accepting a different answer?
    – tripleee
    Mar 11, 2018 at 15:55

11 Answers 11

32

You can do this with AWK using variable substitution. We can set a variable in AWK using -v, and then use AWK's gsub function to substitute all occurrences of a regular expression with that variable.

For example, if the file test has the following contents ...

foo
bar
blah _data_and_data_
foo
_data_ foobar _data_ again

... and the Bash variable $DATA is ...

1
2
3
4
5

... then awk -v r=$DATA '{gsub(/_data_/,r)}1' test replaces all occurrences of the regular expression _data_ in the file test with the contents of $DATA, resulting in the following:

foo
bar
blah 1
2
3
4
5and1
2
3
4
5
foo
1
2
3
4
5 foobar 1
2
3
4
5 again
6
  • 6
    Oh my god, this WORKS! I wish I could +1 you several more times, you saved me man! Dec 21, 2013 at 0:11
  • 4
    Hey Kent, this is an old one, but you seem to be missing quotes around $DATA. confirm? Feb 13, 2014 at 1:53
  • 2
    should add quotes indeed: awk -v r="$DATA" '{gsub(/_data_/,r)}1' test Sep 6, 2017 at 18:39
  • 3
    Apparently some (?) Awks will dislike this and say "newline in string" (though the answer without double quotes around DATA will fail in different ways).
    – tripleee
    Mar 11, 2018 at 12:45
  • 1
    For people confused about the seemingly extraneous 1 after the substitution function, see this question. Short version: leave the 1 there.
    – Ti Strga
    Aug 18, 2022 at 17:31
18

I would suggest simply replacing sed with perl command like this:

perl -i.bak -pe 's/_data_/$ENV{"DATA"}/g' mail.tpl 
3
  • 2
    This is the only that worked for me on OSX. All the other sed and awk based approches failed fo me. Dec 1, 2017 at 13:45
  • 4
    This will fail in unsettling ways if DATA contains a slash. But you can export DATA and access it from inside Perl as $ENV{"DATA"} without getting code and data mixed up. So 's/_data_/$ENV{"DATA"}/g' simply.
    – tripleee
    Mar 11, 2018 at 12:22
  • 1
    @anubhava and @tripleee, you saved my day. This is the only solution that worked for me: neither sed nor awk could do the trick. Thanks. Jul 4, 2019 at 14:37
16

If you build your multiple line text with "\n"s, this will work with a simple sed command as:

DATA=`echo ${DATA} | tr '\n' "\\n"`
#now, DATA="line1\nline2\nline3"
sed "s/_data_/${DATA}/" mail.tpl
5
  • 6
    I'm getting sed: -e expression #1, char 32: unterminated 's' command from that method.
    – Cobra_Fast
    Apr 11, 2012 at 14:20
  • 2
    This is not entirely portable; not all sed dialects accept the \n digraph to represent a literal newline. In some other dialects, you will probably need to use a backslash before a literal newline. Other answers on this page show you how to do that in more detail.
    – tripleee
    Jul 6, 2017 at 4:11
  • 11
    Looking at this again, I don't think there is any version of tr which can replace one character with two characters; it's simply the wrong tool for this job. The newlines are lost by accident because you incorrectly fail to quote the argument to echo, but this has multiple other undesirable side effects; and the tr simply does nothing at all here. This should not be the accepted answer.
    – tripleee
    Mar 11, 2018 at 12:26
  • Yes does not work for me either. Try this solution (below) Apr 15, 2019 at 15:26
  • 1
    Given that the shell is Bash, then there's no need for the (broken) tr invocation - just use ${DATA//$'\n'/\\n}. Aug 14, 2019 at 13:08
15

ring bearer's answer didn't work for me; I think the usage of tr there is wrong, and the way it's written, it simply strips away newlines by use of echo.

Instead, I used sed. I used code from another answer to replace newlines (credit: Zsolt Botykai). I also expected some dollar signs ($) in my input so I took care of that too. You might need to add other input handling. Note the use of double quotes in echo to preserve newlines.

DATA="$(cat whatever)"
ESCAPED_DATA="$(echo "${DATA}" | sed ':a;N;$!ba;s/\n/\\n/g' | sed 's/\$/\\$/g')"

Then you can use ${ESCAPED_DATA} in sed:

cat input | sed 's/one liner/'"${ESCAPED_DATA}"'/' > output 

Just thought I'd share.

1
  • This works well also when multi-line strings have been built in a bash script with $'\n' construct (the cat in the beginning is, of course, then not needed since the string is already in a variable).
    – Ville
    Jul 30, 2017 at 21:43
13

I tried it and sed 's/pattern/\na\nb\nc/g' but it does not work on all systems. What does work is putting a \ followed by a newline in the replace pattern, like this:

sed 's/pattern/a\
b\
c/g'

This appends a line containing b and a line containing c when the pattern is seen.

To put it in a variable, use double backslashes:

export DATA="\\
a\\
b\\
c"

and then:

sed "s/pattern/${DATA}/g"

Note the double quotes.

8

Echo variable into temporary text file.

Insert text file into mail.tpl and delete data from mail.tpl

echo ${DATA} > temp.txt    
sed -i -e "/_data_/r temp.txt" -e "//d" mail.tpl
1
  • 1
    This is the only one that worked for me after trying many other solutions.
    – Kevin
    Oct 6, 2020 at 21:43
4

Escaping all the newlines with a \ (except the last one) worked for me. The last newline must not be escaped not to break the s command.

Example :

DATA="a
b
c"

ESCAPED=$(echo "${DATA}" | sed '$!s@$@\\@g')
echo "${ESCAPED}" 
a\
b\
c

sed "s/pattern/${ESCAPED}/" file
1
  • I had to use double slashes.
    – Stonecraft
    Oct 4, 2021 at 1:40
3

As per the gnu documentation about multiline techniques:

sed '/./{H;$!d;}; x; s/REGEXP/REPLACEMENT/'

The first expression, /./{H;$!d;} operates on all non-empty lines, and adds the current line (in the pattern space) to the hold space. On all lines except the last, the pattern space is deleted and the cycle is restarted.

The x command fetches the accumulated lines from the hold space back to the pattern space.

The s/// command then operates on all the text in the paragraph (including the embedded newlines).

2

You can put your data in a temp file and run:

$ sed '/_data_/r DATA_FILE' mail.tpl | sed '/_data_/d'> temp; mv temp mail.tpl
0
1

Not sure if you have tried to put "\n" in the replace part

sed 's/[pattern]/\
[line 1]\n\
[line 2]\n\
[line n]\n\
/g' mail.tpl

The first line has /\ for readibility reasons. Each line after that is a stand-alone line like you would find in a text editor. Last line is stand-alone, once again for readability reasons. You can make all of this one line if needed. Works on Debian Jessie when I tested it.

1
1
DATA=`echo ${DATA} | tr '\n' "\n"`

sed -z "s/_data_/${DATA}/g" -i mail.tpl

Try this...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.