173

I have the following variable.

echo "|$COMMAND|"

which returns

|
REBOOT|

How can I remove that first newline?

0

12 Answers 12

179

In string, character replacement / deletion

Under , there are some bashisms:

The tr command could be replaced by ${parameter/pattern/string} bashism:

COMMAND=$'\nREBOOT\r   \n'
echo "|${COMMAND}|"
|
   OOT
|

echo "|${COMMAND//[$'\t\r\n']}|"
|REBOOT   |

echo "|${COMMAND//[$'\t\r\n ']}|"
|REBOOT|

See Parameter Expansion and QUOTING in bash's man page:

man -Pless\ +/parameter/pattern/string bash

man -Pless\ +/\/pattern bash
man -Pless\ +/\\\'string\\\' bash

man -Pless\ +/^\\\ *Parameter\\\ Exp bash
man -Pless\ +/^\\\ *QUOTING bash
${parameter//pattern/string}
    If there are two slashes separating parameter and pattern,
    all  matches of pattern are replaced with string.

Further...

As asked by @AlexJordan, this will suppress all specified characters. So what if $COMMAND do contain spaces...

COMMAND=$'         \n        RE BOOT      \r           \n'
echo "|$COMMAND|"
|
           BOOT      
|

read -r COMMAND <<<"${COMMAND//[$'\t\r\n']}"
echo "|$COMMAND|"
|RE BOOT|

Explanation

Answering Vulwsztyn's question:

Why does this work when the pattern is empty?

In ${COMMAND//[$'\t\r\n ']} :

  • The 1st slashe / mean: Pattern substitution
  • The pattern is /[$'\r\n '], begin with / then all matches of pattern are replaced with string
  • Then, the string is empty.

Avoid fork to tr for single string!

Let compare:

COMMAND=$'\nREBOOT\r   \n'
echo ${COMMAND@Q}
$'\nREBOOT\r \n'

COMMAND=$(echo $COMMAND|tr -d '\n\t\r ')
echo ${COMMAND@Q}
'REBOOT'

Then

time for i in {1..1000};do
    COMMAND=$'\nREBOOT\r   \n'
    COMMAND=$(echo $COMMAND|tr -d '\n\t\r ')
done;echo ${COMMAND@Q}

real    0m2.785s
user    0m2.296s
sys     0m0.774s
'REBOOT'

With

COMMAND=$'\nREBOOT\r   \n'
COMMAND="${COMMAND//[$'\t\r\n ']}"
echo ${COMMAND@Q}

time for i in {1..1000};do
    COMMAND=$'\nREBOOT\r   \n'
    COMMAND="${COMMAND//[$'\t\r\n ']}"
done;echo ${COMMAND@Q}

real    0m0.006s
user    0m0.001s
sys     0m0.004s
'REBOOT'

Doing 1'000 forks to tr take more than 2700ms on my host, while same job is done in 6ms ( 464.2x faster!! ) by using built-in bash Parameter Expansion!!

12
  • 1
    This is the way to strip linefeeds from a variable in BASH. Other answers are needlessly invoking an extra TR process. BTW, it has the added benefit of removing ending spaces!
    – ingyhere
    Nov 12, 2015 at 21:06
  • 2
    Note that it also removes internal spaces from a string as well... COMMAND="RE BOOT"; echo "|${COMMAND//[$'\t\r\n ']}|" returns |REBOOT| Mar 16, 2016 at 11:48
  • 2
    @AlexJordan Yes, it's a wanted feature: You could whipe the space after \n to prevent this: COMMAND="RE BOOT"; echo "|${COMMAND//[$'\t\r\n']}|" will return |RE BOOT|. Mar 16, 2016 at 22:01
  • 4
    How is the pattern working in ${COMMAND//[$'\t\r\n']}? I thought ${COMMAND//[\t\r\n]} would simply work, but it didn't. What is the $ symbol for and the single quotes too?
    – haridsv
    May 11, 2017 at 7:43
  • 1
    Regarding the $'' syntax, that's ANSI C quoting (mentioned in wjordans answer).
    – chuckx
    Apr 19, 2018 at 17:55
121

Clean your variable by removing all the linefeeds:

COMMAND=$(echo $COMMAND|tr -d '\n')
5
  • 29
    Doesn't that strip linefeeds? Shouldn't it be tr -d '\r' instead?
    – Izzy
    Nov 17, 2014 at 22:30
  • 6
    Echoing an uncommented variable removes all IFS characters (newline, space, tab by default). So if you're going to do this, you should be aware that all IFS characters are dropped, and you don't need the tr. Simply COMMAND=$(echo $COMMAND) will give a similar effect. Which is, presumably, devil spawn as it invokes a new process, but it's still pretty short and sweet for the human's eyes and if you have a second or two to spare in your life you may be willing to take the hit :-) .
    – Mike S
    Dec 27, 2016 at 22:38
  • 1
    I've updated the question to say "linefeed", as its example did show that it was a linefeed, not a carriage return. This answer is still correct, but should maybe be updated to say "linefeed" instead of "carriage return"? Aug 7, 2018 at 14:14
  • 1
    should have been the accepted answer combined with bash specific ${COMMAND//[$'\t\r\n']}
    – Summer-Sky
    Nov 10, 2021 at 16:36
  • 1
    @MikeS +1 for noticing the antipattern echo $COMMAND (missing quotes). But it's an antipattern and shouldn't be used since it will also expand glob characters if there are any (e.g., *, or [...] or ?). So never use an unquoted expansion, unless your variable is in fact a glob that you want to actually expand. Jan 6 at 10:18
108
echo "|$COMMAND|"|tr '\n' ' '

will replace the newline (in POSIX/Unix it's not a carriage return) with a space.

To be honest I would think about switching away from bash to something more sane though. Or avoiding generating this malformed data in the first place.

Hmmm, this seems like it could be a horrible security hole as well, depending on where the data is coming from.

10
  • The date is coming from a curl request which is on the same server, How would i go about putting that into a new var ? newvar=$(echo "|$COMMAND|"|tr '\n' ' ') Oct 13, 2013 at 13:46
  • 2
    Yes. But please tell me you're not allowing arbitrary people to reboot your server remotely without a password... Oct 13, 2013 at 13:49
  • 50
    Why did you not use tr -d '\n' for dropping instead of replace by a space Oct 13, 2013 at 16:20
  • 4
    Please whipe useless pipes! Use tr '\n' ' ' <<< "|$COMMAND|" instead of echo ... | ... Oct 13, 2013 at 16:24
  • 13
    @F.Hauri: or useless tr's: "|${COMMAND//$'\n'}|"
    – rici
    Oct 13, 2013 at 18:39
15

Using bash:

echo "|${COMMAND/$'\n'}|"

(Note that the control character in this question is a 'newline' (\n), not a carriage return (\r); the latter would have output REBOOT| on a single line.)

Explanation

Uses the Bash Shell Parameter Expansion ${parameter/pattern/string}:

The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. [...] If string is null, matches of pattern are deleted and the / following pattern may be omitted.

Also uses the $'' ANSI-C quoting construct to specify a newline as $'\n'. Using a newline directly would work as well, though less pretty:

echo "|${COMMAND/
}|"

Full example

#!/bin/bash
COMMAND="$'\n'REBOOT"
echo "|${COMMAND/$'\n'}|"
# Outputs |REBOOT|

Or, using newlines:

#!/bin/bash
COMMAND="
REBOOT"
echo "|${COMMAND/
}|"
# Outputs |REBOOT|
0
12

Adding answer to show example of stripping multiple characters including \r using tr and using sed. And illustrating using hexdump.

In my case I had found that a command ending with awk print of the last item |awk '{print $2}' in the line included a carriage-return \r as well as quotes.

I used sed 's/["\n\r]//g' to strip both the carriage-return and quotes.

I could also have used tr -d '"\r\n'.

Interesting to note sed -z is needed if one wishes to remove \n line-feed chars.

$ COMMAND=$'\n"REBOOT"\r   \n'

$ echo "$COMMAND" |hexdump -C
00000000  0a 22 52 45 42 4f 4f 54  22 0d 20 20 20 0a 0a     |."REBOOT".   ..|

$ echo "$COMMAND" |tr -d '"\r\n' |hexdump -C
00000000  52 45 42 4f 4f 54 20 20  20                       |REBOOT   |

$ echo "$COMMAND" |sed 's/["\n\r]//g' |hexdump -C
00000000  0a 52 45 42 4f 4f 54 20  20 20 0a 0a              |.REBOOT   ..|

$ echo "$COMMAND" |sed -z 's/["\n\r]//g' |hexdump -C
00000000  52 45 42 4f 4f 54 20 20  20                       |REBOOT   |

And this is relevant: What are carriage return, linefeed, and form feed?

  • CR == \r == 0x0d
  • LF == \n == 0x0a
10

What worked for me was echo $testVar | tr "\n" " "

Where testVar contained my variable/script-output

5

If you are using bash with the extglob option enabled, you can remove just the trailing whitespace via:

shopt -s extglob
COMMAND=$'\nRE BOOT\r   \n'
echo "|${COMMAND%%*([$'\t\r\n '])}|"

This outputs:

|
RE BOOT|

Or replace %% with ## to replace just the leading whitespace.

1
  • 1
    This worked like a charm with some weird output I got from a docker command. Much thanks!
    – develCuy
    Jul 31, 2018 at 5:54
4

You can simply use echo -n "|$COMMAND|".

$ man echo

-n do not output the trailing newline

4
  • 1
    But at least one of the newlines under concern was not at the end. So this doesn't help.
    – Mike S
    Jan 6 at 23:01
  • With all due respect: have you already tried my solution and checked whether or not it solved the original question?
    – Paulo
    Jan 7 at 17:53
  • No, because the man page is quite clear: -n does not output the trailing newline. That's it, no more no less. If there's an embedded newline- which there is in the original question- then echo -n won't help. I did try it just now, and it behaves exactly as described. Embedded newlines remain; it does not "remove that first newline" which is what the original question asked. In bash, btw.
    – Mike S
    Jan 7 at 18:01
  • Well, I got your point. When I run the command I suggested, the output is the same when you remove the first, even though the command I suggested is not intended for that. I believe I misunderstood the question.
    – Paulo
    Jan 9 at 14:46
4

Use this bashism if you don't want to spawn processes like (tr, sed or awk) for such a simple task. Bash can do that alone:

COMMAND=${COMMAND//$'\n'/}

From the documentation:

${FOO//from/to} Replace all
${FOO/from/to}  Replace first match
4
  • You're posting 9 years after the answer was posted with those very bashisms, with excellent examples, with awesome pointers to the documentation. F. Hauri had the best answer in 2013, and this one adds nothing to it while leaving out some nice extras. Sorry, -1.
    – Mike S
    Jan 6 at 23:07
  • @MikeS Sorry, if we want to write effective bash scripts, it is important to use the least possible amount of external binary executions. That means, we need to handle things with the internal bash meachnisms as possible. This is the first and only answer what does it. Many active bash users do not see any bad in the external calls, others yes. This dualism is very characteristic also on the Unix SE. If you do not have a strong affection for calling the most possible external binaries, I think this answer is not so bad even on your opinion.
    – peterh
    Feb 22 at 15:26
  • @peterh do you not see F. Hauri's answer? He uses the same construct. For example: CLEANED=${COMMAND//[$'\t\r\n']} . I didn't downvote because of its effectiveness in writing bash scripts, I downvoted because it shows up before F. Hauri's answer, which is more complete and includes pointers to the man pages and so on. It adds nothing new.
    – Mike S
    Feb 28 at 17:58
  • @MikeS Well I overlooked it because its complexity. I searched for a quick solution only with internal commands. It is good to you if you can make a living by shell scripts.
    – peterh
    Mar 1 at 6:08
2

To address one possible root of the actual issue, there is a chance you are sourcing a crlf file.

CRLF Example:

.env (crlf)

VARIABLE_A="abc"
VARIABLE_B="def"

run.sh

#!/bin/bash
source .env
echo "$VARIABLE_A"
echo "$VARIABLE_B"
echo "$VARIABLE_A $VARIABLE_B"

Returns:

abc
def
 def

If however you convert to LF:

.env (lf)

VARIABLE_A="abc"
VARIABLE_B="def"

run.sh

#!/bin/bash
source .env
echo "$VARIABLE_A"
echo "$VARIABLE_B"
echo "$VARIABLE_A $VARIABLE_B"

Returns:

abc
def
abc def
1
  • 1
    Thank you for this, I've been pulling my hairs on this for a while wondering why variable substitution was messing my string. Was using a CRLF file on linux... Mar 16, 2021 at 10:21
0

Note that the shell already does that for you if you pass $COMMAND as a parameter instead of a string.

So you can try this:

COMMAND="a
   b
   c"

echo $COMMAND

To add the pipes, you can just use the quotes and no spaces like so:

echo "|"$COMMAND"|"

works just fine and it's a tad simpler than the other solutions. It also works with just /bin/sh.

-2

Grep can be used to filter out any blank lines. This is useful if you have several lines of data and want to remove any empty.

echo "$COMMAND" | grep .

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.