Format that can be reused as shell input
Edit February 2021: bash ${var@Q}
Under Bash, you could store your variable content with Parameter Expansion's @
command for Parameter transformation:
${parameter@operator}
Parameter transformation. The expansion is either a > transforma‐
tion of the value of parameter or information about parameter
itself, depending on the value of operator. Each operator is a
single letter:
Q The expansion is a string that is the value of parameter
quoted in a format that can be reused as input.
...
A The expansion is a string in the form of an assignment
statement or declare command that, if evaluated, will
recreate parameter with its attributes and value.
Sample:
$ var=$'Hello\nGood world.\n'
$ echo "$var"
Hello
Good world.
$ echo "${var@Q}"
$'Hello\nGood world.\n'
$ echo "${var@A}"
var=$'Hello\nGood world.\n'
Old answer
There is a special printf
format directive (%q
) built for this kind of request:
printf [-v var] format [arguments]
%q causes printf to output the corresponding argument
in a format that can be reused as shell input.
Some samples:
read foo
Hello world
printf "%q\n" "$foo"
Hello\ world
printf "%q\n" $'Hello world!\n'
$'Hello world!\n'
This could be used through variables too:
printf -v var "%q" "$foo
"
echo "$var"
$'Hello world\n'
Quick check with all (128) ASCII bytes:
Note that all bytes from 128 to 255 have to be escaped.
This little loop will print all characters from 0x00
to 0x7f
,
by using both: printf %q
and ${var@Q}
method.
for i in {0..127};do
printf -v var %02X $i
printf -v var %b \\x$var
sign=E
printf -v res %q "$var"
[[ $var == "$res" ]] && sign=-
printf '%02X %s %-7s %-7s\n' $i $sign "$res" "${var@Q}"
done |
column
This should render something like:
00 E '' '' 2B - + '+' 56 - V 'V'
01 E $'\001' $'\001' 2C E \, ',' 57 - W 'W'
02 E $'\002' $'\002' 2D - - '-' 58 - X 'X'
03 E $'\003' $'\003' 2E - . '.' 59 - Y 'Y'
04 E $'\004' $'\004' 2F - / '/' 5A - Z 'Z'
05 E $'\005' $'\005' 30 - 0 '0' 5B E \[ '['
06 E $'\006' $'\006' 31 - 1 '1' 5C E \\ '\'
07 E $'\a' $'\a' 32 - 2 '2' 5D E \] ']'
08 E $'\b' $'\b' 33 - 3 '3' 5E E \^ '^'
09 E $'\t' $'\t' 34 - 4 '4' 5F - _ '_'
0A E $'\n' $'\n' 35 - 5 '5' 60 E \` '`'
0B E $'\v' $'\v' 36 - 6 '6' 61 - a 'a'
0C E $'\f' $'\f' 37 - 7 '7' 62 - b 'b'
0D E $'\r' $'\r' 38 - 8 '8' 63 - c 'c'
0E E $'\016' $'\016' 39 - 9 '9' 64 - d 'd'
0F E $'\017' $'\017' 3A - : ':' 65 - e 'e'
10 E $'\020' $'\020' 3B E \; ';' 66 - f 'f'
11 E $'\021' $'\021' 3C E \< '<' 67 - g 'g'
12 E $'\022' $'\022' 3D - = '=' 68 - h 'h'
13 E $'\023' $'\023' 3E E \> '>' 69 - i 'i'
14 E $'\024' $'\024' 3F E \? '?' 6A - j 'j'
15 E $'\025' $'\025' 40 - @ '@' 6B - k 'k'
16 E $'\026' $'\026' 41 - A 'A' 6C - l 'l'
17 E $'\027' $'\027' 42 - B 'B' 6D - m 'm'
18 E $'\030' $'\030' 43 - C 'C' 6E - n 'n'
19 E $'\031' $'\031' 44 - D 'D' 6F - o 'o'
1A E $'\032' $'\032' 45 - E 'E' 70 - p 'p'
1B E $'\E' $'\E' 46 - F 'F' 71 - q 'q'
1C E $'\034' $'\034' 47 - G 'G' 72 - r 'r'
1D E $'\035' $'\035' 48 - H 'H' 73 - s 's'
1E E $'\036' $'\036' 49 - I 'I' 74 - t 't'
1F E $'\037' $'\037' 4A - J 'J' 75 - u 'u'
20 E \ ' ' 4B - K 'K' 76 - v 'v'
21 E \! '!' 4C - L 'L' 77 - w 'w'
22 E \" '"' 4D - M 'M' 78 - x 'x'
23 E \# '#' 4E - N 'N' 79 - y 'y'
24 E \$ '$' 4F - O 'O' 7A - z 'z'
25 - % '%' 50 - P 'P' 7B E \{ '{'
26 E \& '&' 51 - Q 'Q' 7C E \| '|'
27 E \' \' 52 - R 'R' 7D E \} '}'
28 E \( '(' 53 - S 'S' 7E E \~ '~'
29 E \) ')' 54 - T 'T' 7F E $'\177' $'\177'
2A E \* '*' 55 - U 'U'
Where first field is hexadecimal value of byte, second contain E
if character need to be escaped and third field show escaped presentation of character.
Small script looking for limited bunch of characters
For fun, here is another way for looping over a string, grouping all characters by the need to be escaped.
quickListOfSpecialCharsFromString() {
local {q,}char bunch{_0,_1} \
special="${1:-'\`\"/\!@#\$%^&*()-_+={\}[]|;:,.<>? '}"
while IFS= LANG=C LC_ALL=C read -d '' -rn 1 char; do
printf -v qchar %q "$char"
[[ $char == "$qchar" ]]
local -n bunch=bunch_$?
bunch+=(${char@Q})
done < <(printf %s "$special");
printf 'Characters who %sneed to be escaped:\n%s\n' \
"doesn't " "${bunch_0[*]}" "" "${bunch_1[*]}"
}
quickListOfSpecialCharsFromString $'`!@#$%^&*()-_+={}|[]\\;\':",.<>?/ '
Characters who doesn't need to be escaped:
'@' '%' '-' '_' '+' '=' ':' '.' '/'
Characters who need to be escaped:
'`' '!' '#' '$' '^' '&' '*' '(' ')' '{' '}' '|' '[' ']' '\' ';' \' '"' ',' '<' '>' '?' ' '
Why ,
?
You could see some characters that don't always need to be escaped, like ,
, }
and {
.
So not always but sometime:
echo test 1, 2, 3 and 4,5.
test 1, 2, 3 and 4,5.
or
echo test { 1, 2, 3 }
test { 1, 2, 3 }
but care:
echo test{1,2,3}
test1 test2 test3
echo test\ {1,2,3}
test 1 test 2 test 3
echo test\ {\ 1,\ 2,\ 3\ }
test 1 test 2 test 3
echo test\ {\ 1\,\ 2,\ 3\ }
test 1, 2 test 3
See Brace Expansion chapter in bash's man page:
man -P'less +/Brace\ Expansion' bash
Note about percent sign %
.
No, percent sign don't need to be escaped in any posix shell compatible!
But in cron's crontab
!! This is a common issue as syntax used in crontab
is mostly posix shell compatible. but man -Pless\ +/% 5 crontab
:
... Percent-signs (%) in the command, unless escaped with backslash (), will be changed into newline characters...
If you try to use timestamp is crontab, you have to escape %
:
* * * * * echo one more line >>file-$(date +\%F).log