313

Is there any comprehensive list of characters that need to be escaped in Bash? Can it be checked just with sed?

In particular, I was checking whether % needs to be escaped or not. I tried

echo "h%h" | sed 's/%/i/g'

and worked fine, without escaping %. Does it mean % does not need to be escaped? Was this a good way to check the necessity?

And more general: are they the same characters to escape in shell and bash?

0

7 Answers 7

388

There are two easy and safe rules which work not only in sh but also bash.

1. Put the whole string in single quotes

This works for all chars except single quote itself. To escape the single quote, close the quoting before it, insert the single quote, and re-open the quoting.

'I'\''m a s@fe $tring which ends in newline
'

sed command: sed -e "s/'/'\\\\''/g; 1s/^/'/; \$s/\$/'/"

2. Escape every char with a backslash

This works for all characters except newline. For newline characters use single or double quotes. Empty strings must still be handled - replace with ""

\I\'\m\ \a\ \s\@\f\e\ \$\t\r\i\n\g\ \w\h\i\c\h\ \e\n\d\s\ \i\n\ \n\e\w\l\i\n\e"
"

sed command: sed -e 's/./\\&/g; 1{$s/^$/""/}; 1!s/^/"/; $!s/$/"/'.

2b. More readable version of 2

There's an easy safe set of characters, like [a-zA-Z0-9,._+:@%/-], which can be left unescaped to keep it more readable

I\'m\ a\ s@fe\ \$tring\ which\ ends\ in\ newline"
"

sed command: LC_ALL=C sed -e 's/[^a-zA-Z0-9,._+@%/-]/\\&/g; 1{$s/^$/""/}; 1!s/^/"/; $!s/$/"/'.


Note that in a sed program, one can't know whether the last line of input ends with a newline byte (except when it's empty). That's why both above sed commands assume it does not. You can add a quoted newline manually.

Note that shell variables are only defined for text in the POSIX sense. Processing binary data is not defined. For the implementations that matter, binary works with the exception of NUL bytes (because variables are implemented with C strings, and meant to be used as C strings, namely program arguments), but you should switch to a "binary" locale such as latin1.


(You can easily validate the rules by reading the POSIX spec for sh. For bash, check the reference manual linked by @AustinPhillips)

32
  • 2
    Note: a good variation on #1 can bee seen here: github.com/scop/bash-completion/blob/…. It does not require running sed, but does require bash.
    – jwd
    Commented Feb 10, 2017 at 18:52
  • 7
    Note for anyone else (like me!) who struggles to get these working.... looks like the flavour of sed you get on OSX doesn't run these sed commands properly. They work fine on Linux though!
    – dalelane
    Commented Jun 21, 2017 at 21:58
  • @dalelane: Can't test here. Please edit when you have a version that works on both.
    – Jo So
    Commented Jun 22, 2017 at 14:34
  • Seems you missed-out should the string start with a '-' (minus), or does that only apply to filenames? - in latter case need a './' in front.
    – slashmais
    Commented Aug 15, 2017 at 9:42
  • 1
    For macOS users without GNU sed: @fd0 has a sed option to escape every character: apple.stackexchange.com/a/363400/409134 And I wrote a solution that only escapes the control characters using perl: apple.stackexchange.com/a/458279/409134
    – Nils
    Commented Apr 8, 2023 at 14:02
108

Format that can be reused as shell input

Edit February 2021: ${var@Q}

Under Bash, you could store your variable content with Parameter Expansion's @ command for Parameter transformation:

${parameter@operator}
       Parameter transformation.  The expansion is either a > transforma‐
       tion of the value of parameter or  information  about  parameter
       itself,  depending on the value of operator.  Each operator is a
       single letter:

       Q      The expansion is a string that is the value of  parameter
              quoted in a format that can be reused as input.
...
       A      The  expansion  is  a string in the form of an assignment
              statement or declare command  that,  if  evaluated,  will
              recreate parameter with its attributes and value.

Sample:

$ var=$'Hello\nGood world.\n'
$ echo "$var"
Hello
Good world.

$ echo "${var@Q}"
$'Hello\nGood world.\n'

$ echo "${var@A}"
var=$'Hello\nGood world.\n'

Old answer

There is a special printf format directive (%q) built for this kind of request:

printf [-v var] format [arguments]

    %q     causes printf to output the corresponding argument
           in a format that can be reused as shell input.

Some samples:

read foo
Hello world
printf "%q\n" "$foo"
Hello\ world

printf "%q\n" $'Hello world!\n'
$'Hello world!\n'

This could be used through variables too:

printf -v var "%q" "$foo
"
echo "$var"
$'Hello world\n'

Quick check with all (128) ASCII bytes:

Note that all bytes from 128 to 255 have to be escaped.

This little loop will print all characters from 0x00 to 0x7f, by using both: printf %q and ${var@Q} method.

for i in {0..127}; do
    printf -v var %02X $i
    printf -v var %b \\x$var
    sign=E
    printf -v res %q "$var"
    [[ $var == "$res" ]] && sign=-
    printf '%02X %s %-*s %-*s\n' $i $sign $(( 31 < i && i < 96 ? 2 : 8
        )) "${res}" $(( 31 < i && i < 96 ? 3 : 8)) "${var@Q}"
done |
    pr -w100 --sep-string='' -t4 |
    sed 's/\o11\+/\o11/g'

This should render something like:

00 E ''       ''         20 E \  ' '      40 - @  '@'      60 E \`       '`'
01 E $'\001'  $'\001'    21 E \! '!'      41 - A  'A'      61 - a        'a'
02 E $'\002'  $'\002'    22 E \" '"'      42 - B  'B'      62 - b        'b'
03 E $'\003'  $'\003'    23 E \# '#'      43 - C  'C'      63 - c        'c'
04 E $'\004'  $'\004'    24 E \$ '$'      44 - D  'D'      64 - d        'd'
05 E $'\005'  $'\005'    25 - %  '%'      45 - E  'E'      65 - e        'e'
06 E $'\006'  $'\006'    26 E \& '&'      46 - F  'F'      66 - f        'f'
07 E $'\a'    $'\a'      27 E \' \'       47 - G  'G'      67 - g        'g'
08 E $'\b'    $'\b'      28 E \( '('      48 - H  'H'      68 - h        'h'
09 E $'\t'    $'\t'      29 E \) ')'      49 - I  'I'      69 - i        'i'
0A E $'\n'    $'\n'      2A E \* '*'      4A - J  'J'      6A - j        'j'
0B E $'\v'    $'\v'      2B - +  '+'      4B - K  'K'      6B - k        'k'
0C E $'\f'    $'\f'      2C E \, ','      4C - L  'L'      6C - l        'l'
0D E $'\r'    $'\r'      2D - -  '-'      4D - M  'M'      6D - m        'm'
0E E $'\016'  $'\016'    2E - .  '.'      4E - N  'N'      6E - n        'n'
0F E $'\017'  $'\017'    2F - /  '/'      4F - O  'O'      6F - o        'o'
10 E $'\020'  $'\020'    30 - 0  '0'      50 - P  'P'      70 - p        'p'
11 E $'\021'  $'\021'    31 - 1  '1'      51 - Q  'Q'      71 - q        'q'
12 E $'\022'  $'\022'    32 - 2  '2'      52 - R  'R'      72 - r        'r'
13 E $'\023'  $'\023'    33 - 3  '3'      53 - S  'S'      73 - s        's'
14 E $'\024'  $'\024'    34 - 4  '4'      54 - T  'T'      74 - t        't'
15 E $'\025'  $'\025'    35 - 5  '5'      55 - U  'U'      75 - u        'u'
16 E $'\026'  $'\026'    36 - 6  '6'      56 - V  'V'      76 - v        'v'
17 E $'\027'  $'\027'    37 - 7  '7'      57 - W  'W'      77 - w        'w'
18 E $'\030'  $'\030'    38 - 8  '8'      58 - X  'X'      78 - x        'x'
19 E $'\031'  $'\031'    39 - 9  '9'      59 - Y  'Y'      79 - y        'y'
1A E $'\032'  $'\032'    3A - :  ':'      5A - Z  'Z'      7A - z        'z'
1B E $'\E'    $'\E'      3B E \; ';'      5B E \[ '['      7B E \{       '{'
1C E $'\034'  $'\034'    3C E \< '<'      5C E \\ '\'      7C E \|       '|'
1D E $'\035'  $'\035'    3D - =  '='      5D E \] ']'      7D E \}       '}'
1E E $'\036'  $'\036'    3E E \> '>'      5E E \^ '^'      7E E \~       '~'
1F E $'\037'  $'\037'    3F E \? '?'      5F - _  '_'      7F E $'\177'  $'\177'

Where

  • first field is hexadecimal value of byte,
  • second contain E if character need to be escaped,
  • third field show escaped presentation of character and
  • last field show useable version printed by ${var@Q} syntax.

Small function looking for limited bunch of characters

For fun, here is another way for looping over a string, grouping all characters by the need to be escaped.

specialCharsFromString() {
    local {q,}char bunch{_0,_1} \
        special="${1:-'\`\"/\!@#\$%^&*()-_+={\}[]|;:,.<>? '}"
    while IFS= LANG=C LC_ALL=C read -d '' -rn 1 char; do
        printf -v qchar %q "$char"
        [[ $char == "$qchar" ]]
        local -n bunch=bunch_$?
        bunch+=(${char@Q})
    done < <(printf %s "$special");
    printf 'Characters who %sneed to be escaped:\n%s\n' \
        "doesn't " "${bunch_0[*]}" "" "${bunch_1[*]}"
}
specialCharsFromString $'`!@#$%^&*()-_+={}|[]\\;\':",.<>?/ '
Characters who doesn't need to be escaped:
'@' '%' '-' '_' '+' '=' ':' '.' '/'
Characters who need to be escaped:
'`' '!' '#' '$' '^' '&' '*' '(' ')' '{' '}' '|' '[' ']' '\' ';' \' '"' ',' '<' '>' '?' ' '

Why ,?

You could see some characters that don't always need to be escaped, like ,, } and {.

So not always but sometime:

echo test 1, 2, 3 and 4,5.
test 1, 2, 3 and 4,5.

or

echo test { 1, 2, 3 }
test { 1, 2, 3 }

but care:

echo test{1,2,3}
test1 test2 test3

echo test\ {1,2,3}
test 1 test 2 test 3

echo test\ {\ 1,\ 2,\ 3\ }
test  1 test  2 test  3

echo test\ {\ 1\,\ 2,\ 3\ }
test  1, 2 test  3

See Brace Expansion chapter in 's man page:

  man -P'less +/Brace\ Expansion' bash

Note about percent sign %.

No, percent sign don't need to be escaped in any compatible!

But in 's crontab!! This is a common issue as syntax used in crontab is mostly compatible. but man -Pless\ +/% 5 crontab:

... Percent-signs (%) in the command, unless escaped with backslash (), will be changed into newline characters...

If you try to use timestamp is crontab, you have to escape %:

* * * * * echo one more line >>file-$(date +\%F).log

Note about dashes sign -.

There is no reason to escape a dash in a string except if you try to use them at begin of string as 1st argument of a regular command:

printf '- %3d %s\n' $((count++)) "Some String"
bash: printf: - : invalid option

ls -dashedFilename
ls: invalid option -- 'e'

Even escaped, this wont go better:

ls \-dashedFilename
ls: invalid option -- 'e'

For this, the recommended way is to use double dash --:

From POSIX.1-2017 Utility Conventions Guideline 10

Guideline 10:

The first -- argument that is not an option-argument should be accepted as a delimiter indicating the end of options. Any following arguments should be treated as operands, even if they begin with the - character.

printf  -- '- %3d %s\n' $((count++)) "Some String"
-   2 Some String

ls -- -dashedFilename
ls: cannot access '-dashedFilename': No such file or directory

Ok, they don't exist.

Alternative: Simply avoid dashes at begin of any string:

printf '\55 %3d %s\n' $((count++)) "Some other String"
-   3 Some other String
printf '\x2d %3d %s\n' $((count++)) "Some other String"
-   4 Some other String

Using any of octal or hexadecimal representation of dash

ls ./-dashedFilename
ls: cannot access './-dashedFilename': No such file or directory

Using relative or full path.

12
  • 2
    @ThorSummoner, not if you pass the string as a literal argument to the shell from a different language (where you presumably already know how to quote). In Python: subprocess.Popen(['bash', '-c', 'printf "%q\0" "$@"', '_', arbitrary_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE).communicate() will give you a properly shell-quoted version of arbitrary_string. Commented Jul 16, 2015 at 23:00
  • 2
    FYI bash's %q was broken for a long time - If my mind serves me well, an error was fixed (but might still be broken) in 2013 after being broken for ~10 years. So don't rely on it.
    – Jo So
    Commented Feb 3, 2017 at 17:36
  • 2
    Thank you to add special notes about ,. I was surprised to learn that built-in Bash printf -- %q ',' gives \,, but /usr/bin/printf -- %q ',' gives , (un-escapted). Same for other chars: {, |, }, ~.
    – kevinarpe
    Commented May 15, 2020 at 14:22
  • 1
    @ilkkachu Depending on local config, using utf or iso as default, playing with bytes between 128 to 255 could lead to strange behaviour Commented Feb 25, 2021 at 10:29
  • 2
    That new @Q is very useful!
    – fedorqui
    Commented Feb 2, 2022 at 9:49
56

Using the print '%q' technique, we can run a loop to find out which characters are special:

#!/bin/bash
special=$'`!@#$%^&*()-_+={}|[]\\;\':",.<>?/ '
for ((i=0; i < ${#special}; i++)); do
    char="${special:i:1}"
    printf -v q_char '%q' "$char"
    if [[ "$char" != "$q_char" ]]; then
        printf 'Yes - character %s needs to be escaped\n' "$char"
    else
        printf 'No - character %s does not need to be escaped\n' "$char"
    fi
done | sort

It gives this output:

No, character % does not need to be escaped
No, character + does not need to be escaped
No, character - does not need to be escaped
No, character . does not need to be escaped
No, character / does not need to be escaped
No, character : does not need to be escaped
No, character = does not need to be escaped
No, character @ does not need to be escaped
No, character _ does not need to be escaped
Yes, character   needs to be escaped
Yes, character ! needs to be escaped
Yes, character " needs to be escaped
Yes, character # needs to be escaped
Yes, character $ needs to be escaped
Yes, character & needs to be escaped
Yes, character ' needs to be escaped
Yes, character ( needs to be escaped
Yes, character ) needs to be escaped
Yes, character * needs to be escaped
Yes, character , needs to be escaped
Yes, character ; needs to be escaped
Yes, character < needs to be escaped
Yes, character > needs to be escaped
Yes, character ? needs to be escaped
Yes, character [ needs to be escaped
Yes, character \ needs to be escaped
Yes, character ] needs to be escaped
Yes, character ^ needs to be escaped
Yes, character ` needs to be escaped
Yes, character { needs to be escaped
Yes, character | needs to be escaped
Yes, character } needs to be escaped

Some of the results, like , look a little suspicious. Would be interesting to get @CharlesDuffy's inputs on this.

2
  • 3
    You may read answer to , look a little suspicious at last paragraph of my answer Commented May 17, 2018 at 19:00
  • 4
    Keep in mind that %q doesn't know where within the shell you are planing to use the character, so it will escape all characters that can have a special meaning in any possible shell context. , itself has no special meaning to she shell but as @F.Hauri has pointed out in his reply, it does have a special meaning within {...} brace expansion: gnu.org/savannah-checkouts/gnu/bash/manual/… This is like ! which also only requires expansion in specific situations, not in general: echo Hello World! works just fine, yet echo test!test will fail.
    – Mecki
    Commented May 17, 2019 at 19:25
52

To save someone else from having to RTFM... in bash:

Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !.

...so if you escape those (and the quote itself, of course) you're probably okay.

If you take a more conservative 'when in doubt, escape it' approach, it should be possible to avoid getting instead characters with special meaning by not escaping identifier characters (i.e. ASCII letters, numbers, or '_'). It's very unlikely these would ever (i.e. in some weird POSIX-ish shell) have special meaning and thus need to be escaped.

2
19

Characters that need escaping are different in Bourne or POSIX shell than Bash. Generally (very) Bash is a superset of those shells, so anything you escape in shell should be escaped in Bash.

A nice general rule would be "if in doubt, escape it". But escaping some characters gives them a special meaning, like \n. These are listed in the man bash pages under Quoting and echo.

Other than that, escape any character that is not alphanumeric, it is safer. I don't know of a single definitive list.

The man pages list them all somewhere, but not in one place. Learn the language, that is the way to be sure.

One that has caught me out is !. This is a special character (history expansion) in Bash (and csh) but not in Korn shell. Even echo "Hello world!" gives problems. Using single-quotes, as usual, removes the special meaning.

3
  • 1
    I specially like the A nice general rule would be "if in doubt, escape it" advice. Still have the doubt whether checking with sed is good enough to see if it has to be escaped. Thanks for your answer!
    – fedorqui
    Commented Apr 4, 2013 at 15:29
  • 2
    @fedorqui: Checking with sed is not necessary, you could check with almost anything. sed is not the issue, bash is. Inside single quotes there are no special characters (except single quotes), you can't even escape characters there. A sed command should usually be inside single quotes because RE metacharacters have too many overlaps with shell metacharacters to be safe. The exception is when embedding shell variables, which has to be done carefully.
    – cdarke
    Commented Apr 5, 2013 at 9:11
  • 5
    Check with echo. If you get out what you put in, it doesn't need to be escaped. :)
    – Mark Reed
    Commented Jul 28, 2014 at 15:57
7

I presume that you're talking about bash strings. There are different types of strings which have a different set of requirements for escaping. eg. Single quotes strings are different from double quoted strings.

The best reference is the Quoting section of the bash manual.

It explains which characters needs escaping. Note that some characters may need escaping depending on which options are enabled such as history expansion.

4
  • 3
    So it confirms that escaping is such a jungle without an easy solution, will have to check each case. Thanks!
    – fedorqui
    Commented Apr 4, 2013 at 15:30
  • @fedorqui As with any language, there's a set of rules to be followed. For bash string escaping, the set of rules is quite small as described in the manual. The easiest string to use is single quotes since nothing needs escaping. However, there is no way to include a single quote in a single quoted string. Commented Apr 4, 2013 at 22:11
  • @fedorqui. It's not a jungle. Escaping is quite doable. See my new post.
    – Jo So
    Commented Nov 18, 2013 at 16:47
  • @fedorqui You can't use a single quote inside a single-quoted string but you can "escape" it with something like: 'text'"'"'more text'
    – CR.
    Commented Nov 25, 2014 at 3:22
7

I noticed that bash automatically escapes some characters when using auto-complete.

For example, if you have a directory named dir:A, bash will auto-complete to dir\:A

Using this, I runned some experiments using characters of the ASCII table and derived the following lists:

Characters that bash escapes on auto-complete: (includes space)

 !"$&'()*,:;<=>?@[\]^`{|}

Characters that bash does not escape:

#%+-.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~

(I excluded /, as it cannot be used in directory names)

3
  • 3
    If you really wanted to have a comprehensive list, I'd suggest looking at which characters printf %q does and does not modify if passed as an argument -- ideally, going through the entire characterset. Commented Jan 30, 2016 at 4:23
  • There are instances where even with the apostrophe string, you may wish to escape letters and numbers to produce special-characters. For example: tr '\n' '\t' which translates newline characters into tab characters. Commented Aug 9, 2016 at 17:54
  • @CharlesDuffy The characters that auto-complete escapes are somewhat different from what printf %q does, I ran into this testing a pathname containing the 'home' tilde (which %q escapes, causing a problem for me, where auto-complete does not).
    – Compholio
    Commented Nov 3, 2021 at 19:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.