Format that can be reused as shell input
Edit February 2021: bash ${var@Q}
Under Bash, you could store your variable content with Parameter Expansion's @
command for Parameter transformation:
${parameter@operator}
Parameter transformation. The expansion is either a > transforma‐
tion of the value of parameter or information about parameter
itself, depending on the value of operator. Each operator is a
single letter:
Q The expansion is a string that is the value of parameter
quoted in a format that can be reused as input.
...
A The expansion is a string in the form of an assignment
statement or declare command that, if evaluated, will
recreate parameter with its attributes and value.
Sample:
$ var=$'Hello\nGood world.\n'
$ echo "$var"
Hello
Good world.
$ echo "${var@Q}"
$'Hello\nGood world.\n'
$ echo "${var@A}"
var=$'Hello\nGood world.\n'
Old answer
There is a special printf
format directive (%q
) built for this kind of request:
printf [-v var] format [arguments]
%q causes printf to output the corresponding argument
in a format that can be reused as shell input.
Some samples:
read foo
Hello world
printf "%q\n" "$foo"
Hello\ world
printf "%q\n" $'Hello world!\n'
$'Hello world!\n'
This could be used through variables too:
printf -v var "%q" "$foo
"
echo "$var"
$'Hello world\n'
Quick check with all (128) ASCII bytes:
Note that all bytes from 128 to 255 have to be escaped.
This little loop will print all characters from 0x00
to 0x7f
,
by using both: printf %q
and ${var@Q}
method.
for i in {0..127}; do
printf -v var %02X $i
printf -v var %b \\x$var
sign=E
printf -v res %q "$var"
[[ $var == "$res" ]] && sign=-
printf '%02X %s %-*s %-*s\n' $i $sign $(( 31 < i && i < 96 ? 2 : 8
)) "${res}" $(( 31 < i && i < 96 ? 3 : 8)) "${var@Q}"
done |
pr -w100 --sep-string='' -t4 |
sed 's/\o11\+/\o11/g'
This should render something like:
00 E '' '' 20 E \ ' ' 40 - @ '@' 60 E \` '`'
01 E $'\001' $'\001' 21 E \! '!' 41 - A 'A' 61 - a 'a'
02 E $'\002' $'\002' 22 E \" '"' 42 - B 'B' 62 - b 'b'
03 E $'\003' $'\003' 23 E \# '#' 43 - C 'C' 63 - c 'c'
04 E $'\004' $'\004' 24 E \$ '$' 44 - D 'D' 64 - d 'd'
05 E $'\005' $'\005' 25 - % '%' 45 - E 'E' 65 - e 'e'
06 E $'\006' $'\006' 26 E \& '&' 46 - F 'F' 66 - f 'f'
07 E $'\a' $'\a' 27 E \' \' 47 - G 'G' 67 - g 'g'
08 E $'\b' $'\b' 28 E \( '(' 48 - H 'H' 68 - h 'h'
09 E $'\t' $'\t' 29 E \) ')' 49 - I 'I' 69 - i 'i'
0A E $'\n' $'\n' 2A E \* '*' 4A - J 'J' 6A - j 'j'
0B E $'\v' $'\v' 2B - + '+' 4B - K 'K' 6B - k 'k'
0C E $'\f' $'\f' 2C E \, ',' 4C - L 'L' 6C - l 'l'
0D E $'\r' $'\r' 2D - - '-' 4D - M 'M' 6D - m 'm'
0E E $'\016' $'\016' 2E - . '.' 4E - N 'N' 6E - n 'n'
0F E $'\017' $'\017' 2F - / '/' 4F - O 'O' 6F - o 'o'
10 E $'\020' $'\020' 30 - 0 '0' 50 - P 'P' 70 - p 'p'
11 E $'\021' $'\021' 31 - 1 '1' 51 - Q 'Q' 71 - q 'q'
12 E $'\022' $'\022' 32 - 2 '2' 52 - R 'R' 72 - r 'r'
13 E $'\023' $'\023' 33 - 3 '3' 53 - S 'S' 73 - s 's'
14 E $'\024' $'\024' 34 - 4 '4' 54 - T 'T' 74 - t 't'
15 E $'\025' $'\025' 35 - 5 '5' 55 - U 'U' 75 - u 'u'
16 E $'\026' $'\026' 36 - 6 '6' 56 - V 'V' 76 - v 'v'
17 E $'\027' $'\027' 37 - 7 '7' 57 - W 'W' 77 - w 'w'
18 E $'\030' $'\030' 38 - 8 '8' 58 - X 'X' 78 - x 'x'
19 E $'\031' $'\031' 39 - 9 '9' 59 - Y 'Y' 79 - y 'y'
1A E $'\032' $'\032' 3A - : ':' 5A - Z 'Z' 7A - z 'z'
1B E $'\E' $'\E' 3B E \; ';' 5B E \[ '[' 7B E \{ '{'
1C E $'\034' $'\034' 3C E \< '<' 5C E \\ '\' 7C E \| '|'
1D E $'\035' $'\035' 3D - = '=' 5D E \] ']' 7D E \} '}'
1E E $'\036' $'\036' 3E E \> '>' 5E E \^ '^' 7E E \~ '~'
1F E $'\037' $'\037' 3F E \? '?' 5F - _ '_' 7F E $'\177' $'\177'
Where
- first field is hexadecimal value of byte,
- second contain
E
if character need to be escaped,
- third field show escaped presentation of character and
- last field show useable version printed by
${var@Q}
syntax.
Small function looking for limited bunch of characters
For fun, here is another way for looping over a string, grouping all characters by the need to be escaped.
specialCharsFromString() {
local {q,}char bunch{_0,_1} \
special="${1:-'\`\"/\!@#\$%^&*()-_+={\}[]|;:,.<>? '}"
while IFS= LANG=C LC_ALL=C read -d '' -rn 1 char; do
printf -v qchar %q "$char"
[[ $char == "$qchar" ]]
local -n bunch=bunch_$?
bunch+=(${char@Q})
done < <(printf %s "$special");
printf 'Characters who %sneed to be escaped:\n%s\n' \
"doesn't " "${bunch_0[*]}" "" "${bunch_1[*]}"
}
specialCharsFromString $'`!@#$%^&*()-_+={}|[]\\;\':",.<>?/ '
Characters who doesn't need to be escaped:
'@' '%' '-' '_' '+' '=' ':' '.' '/'
Characters who need to be escaped:
'`' '!' '#' '$' '^' '&' '*' '(' ')' '{' '}' '|' '[' ']' '\' ';' \' '"' ',' '<' '>' '?' ' '
Why ,
?
You could see some characters that don't always need to be escaped, like ,
, }
and {
.
So not always but sometime:
echo test 1, 2, 3 and 4,5.
test 1, 2, 3 and 4,5.
or
echo test { 1, 2, 3 }
test { 1, 2, 3 }
but care:
echo test{1,2,3}
test1 test2 test3
echo test\ {1,2,3}
test 1 test 2 test 3
echo test\ {\ 1,\ 2,\ 3\ }
test 1 test 2 test 3
echo test\ {\ 1\,\ 2,\ 3\ }
test 1, 2 test 3
See Brace Expansion chapter in bash's man page:
man -P'less +/Brace\ Expansion' bash
Note about percent sign %
.
No, percent sign don't need to be escaped in any posix shell compatible!
But in cron's crontab
!! This is a common issue as syntax used in crontab
is mostly posix shell compatible. but man -Pless\ +/% 5 crontab
:
... Percent-signs (%) in the command, unless escaped with backslash (), will be changed into newline characters...
If you try to use timestamp is crontab, you have to escape %
:
* * * * * echo one more line >>file-$(date +\%F).log
Note about dashes sign -
.
There is no reason to escape a dash in a string except if you try to use them at begin of string as 1st argument of a regular command:
printf '- %3d %s\n' $((count++)) "Some String"
bash: printf: - : invalid option
ls -dashedFilename
ls: invalid option -- 'e'
Even escaped, this wont go better:
ls \-dashedFilename
ls: invalid option -- 'e'
For this, the recommended way is to use double dash --
:
From POSIX.1-2017 Utility Conventions Guideline 10
Guideline 10:
The first --
argument that is not an option-argument should be accepted as a delimiter indicating the end of options. Any following arguments should be treated as operands, even if they begin with the -
character.
printf -- '- %3d %s\n' $((count++)) "Some String"
- 2 Some String
ls -- -dashedFilename
ls: cannot access '-dashedFilename': No such file or directory
Ok, they don't exist.
Alternative: Simply avoid dashes at begin of any string:
printf '\55 %3d %s\n' $((count++)) "Some other String"
- 3 Some other String
printf '\x2d %3d %s\n' $((count++)) "Some other String"
- 4 Some other String
Using any of octal or hexadecimal representation of dash
ls ./-dashedFilename
ls: cannot access './-dashedFilename': No such file or directory
Using relative or full path.