26

Suppose I have a #!/bin/sh script which can take a variety of positional parameters, some of which may include spaces, either/both kinds of quotes, etc. I want to iterate "$@" and for each argument either process it immediately somehow, or save it for later. At the end of the script I want to launch (perhaps exec) another process, passing in some of these parameters with all special characters intact.

If I were doing no processing on the parameters, othercmd "$@" would work fine, but I need to pull out some parameters and process them a bit.

If I could assume Bash, then I could use printf %q to compute quoted versions of args that I could eval later, but this would not work on e.g. Ubuntu's Dash (/bin/sh).

Is there any equivalent to printf %q that can be written in a plain Bourne shell script, using only built-ins and POSIX-defined utilities, say as a function I could copy into a script?

For example, a script trying to ls its arguments in reverse order:

#!/bin/sh
args=
for arg in "$@"
do
    args="'$arg' $args"
done
eval "ls $args"

works for many cases:

$ ./handle goodbye "cruel world"
ls: cannot access cruel world: No such file or directory
ls: cannot access goodbye: No such file or directory

but not when ' is used:

$ ./handle goodbye "cruel'st world"
./handle: 1: eval: Syntax error: Unterminated quoted string

and the following works fine but relies on Bash:

#!/bin/bash
args=
for arg in "$@"
do
    printf -v argq '%q' "$arg"
    args="$argq $args"
done
eval "ls $args"
2

6 Answers 6

14

This is absolutely doable.

The answer you see by Jesse Glick is approximately there, but it has a couple of bugs, and I have a few more alternatives for your consideration, since this is a problem I ran into more than once.

First, and you might already know this, echo is a bad idea, one should use printf instead, if the goal is portability: "echo" has undefined behavior in POSIX if the argument it receives is "-n", and in practice some implementations of echo treat -n as a special option, while others just treat it as a normal argument to print. So that becomes this:

esceval()
{
    printf %s "$1" | sed "s/'/'\"'\"'/g"
}

Alternatively, instead of escaping embedded single quotes by making them into:

'"'"'

..instead you could turn them into:

'\''

..stylistic differences I guess (I imagine performance difference is negligible either way, though I've never tested). The resulting sed string looks like this:

esceval()
{
    printf %s "$1" | sed "s/'/'\\\\''/g"
}

(It's four backslashes because double quotes swallow two of them, and leaving two, and then sed swallows one, leaving just the one. Personally, I find this way more readable so that's what I'll use in the rest of the examples that involve it, but both should be equivalent.)

BUT, we still have a bug: command substitution will delete at least one (but in many shells ALL) of the trailing newlines from the command output (not all whitespace, just newlines specifically). So the above solution works unless you have newline(s) at the very end of an argument. Then you'll lose that/those newline(s). The fix is obviously simple: Add another character after the actual command value before outputting from your quote/esceval function. Incidentally, we already needed to do that anyway, because we needed to start and stop the escaped argument with single quotes. You have two alternatives:

esceval()
{
    printf '%s\n' "$1" | sed "s/'/'\\\\''/g; 1 s/^/'/; $ s/$/'/"
}

This will ensure the argument comes out already fully escaped, no need for adding more single quotes when building the final string. This is probably the closest thing you will get to a single, inline-able version. If you're okay with having a sed dependency, you can stop here.

If you're not okay with the sed dependency, but you're fine with assuming that your shell is actually POSIX-compliant (there are still some out there, notably the /bin/sh on Solaris 10 and below, which won't be able to do this next variant - but almost all shells you need to care about will do this just fine):

esceval()
{
    printf \'
    unescaped=$1
    while :
    do
        case $unescaped in
        *\'*)
            printf %s "${unescaped%%\'*}""'\''"
            unescaped=${unescaped#*\'}
            ;;
        *)
            printf %s "$unescaped"
            break
        esac
    done
    printf \'
}

You might notice seemingly redundant quoting here:

printf %s "${unescaped%%\'*}""'\''"

..this could be replaced with:

printf %s "${unescaped%%\'*}'\''"

The only reason I do the former, is because one upon a time there were Bourne shells which had bugs when substituting variables into quoted strings where the quote around the variable didn't exactly start and end where the variable substitution did. Hence it's a paranoid portability habit of mine. In practice, you can do the latter, and it won't be a problem.

If you don't want to clobber the variable unescaped in the rest of your shell environment, then you can wrap the entire contents of that function in a subshell, like so:

esceval()
{
  (
    printf \'
    unescaped=$1
    while :
    do
        case $unescaped in
        *\'*)
            printf %s "${unescaped%%\'*}""'\''"
            unescaped=${unescaped#*\'}
            ;;
        *)
            printf %s "$unescaped"
            break
        esac
    done
    printf \'
  )
}

"But wait", you say: "What I want to do this on MULTIPLE arguments in one command? And I want the output to still look kinda nice and legible for me as a user if I run it from the command line for whatever reason."

Never fear, I have you covered:

esceval()
{
    case $# in 0) return 0; esac
    while :
    do
        printf "'"
        printf %s "$1" | sed "s/'/'\\\\''/g"
        shift
        case $# in 0) break; esac
        printf "' "
    done
    printf "'\n"
}

..or the same thing, but with the shell-only version:

esceval()
{
  case $# in 0) return 0; esac
  (
    while :
    do
        printf "'"
        unescaped=$1
        while :
        do
            case $unescaped in
            *\'*)
                printf %s "${unescaped%%\'*}""'\''"
                unescaped=${unescaped#*\'}
                ;;
            *)
                printf %s "$unescaped"
                break
            esac
        done
        shift
        case $# in 0) break; esac
        printf "' "
    done
    printf "'\n"
  )
}

In those last four, you could collapse some of the outer printf statements and roll their single quotes up into another printf - I kept them separate because I feel it makes the logic more clear when you can see the starting and ending single-quotes on separate print statements.

P.S. There's also this monstrosity I made, which is a polyfill which will select between the previous two versions depending on if your shell seems to be capable of supporting the necessary variable substitution syntax (it looks awful though, because the shell-only version has to be inside an eval-ed string to keep the incompatible shells from barfing when they see it): https://github.com/mentalisttraceur/esceval/blob/master/sh/esceval.sh

15
  • 1
    Great stuff, but you need printf "'\\\''" instead of printf "'\''" in the pure shell solution (the version on Github, printf "'"'\''"'", breaks altogether). To make the sed solution multi-line-capable, you need to read all lines up front: esceval(){ printf '%s\n' "$1" | sed -e ':a' -e '$!{N;ba' -e '}' -e "s/'/'\\\\''/g; s/^/'/; s/$/'/"; }. Quibble: It's generally better not to use all-uppercase variable names so as to avoid clashes with environment variables and special shell variables.
    – mklement0
    Commented Apr 25, 2015 at 15:58
  • 1
    Thanks for your thoughtful feedback. Re your revised sed solution: While both yours and mine should be portable, yours is preferable, because it's simpler and doesn't read all lines at once. However, it requires a few tweaks: (a) the 1 ... substitution must be placed AFTER the general substitution, so that the latter doesn't also replace the just-added, initial '; (b) as in my sed solution, \n must be appended to the printf command so as to ensure that trailing newlines are accurately preserved: esceval() { printf '%s\n' "$1" | sed "s/'/'\\\\''/g; 1 s/^/'/; $ s/$/'/"; }
    – mklement0
    Commented Apr 26, 2015 at 14:02
  • 1
    Re printf "'"'\''"'": it actually makes sense to me that that would break, because it contains '\'', which is an attempt to include a single quote inside a single-quoted string, which isn't supported in a POSIX shell: "A single-quote cannot occur within single-quotes."
    – mklement0
    Commented Apr 26, 2015 at 14:19
  • 1
    Re printf "'\''": I think the problem here are variations in printf behavior, not shell string-parsing: if you use printf %s "'\''", all shells should behave the same again (verified in recent versions of bash, dash, ksh, zsh). printf is a builtin most shells, and behavior differs with respect to processing the format string (incidentally, the utility form of printf also differs across platforms); by using %s, you eliminate these variations. Given that, you could even eliminate the separate printf statement and use printf %s "${unescaped%%\'*}'\''" instead.
    – mklement0
    Commented Apr 26, 2015 at 17:09
  • 1
    @mklement0 Yes, it was definitely a printf handling-of-format-string incompatibility rather than bad Bourne shell syntax - anyway, I have fixed all of the examples in my post, and my github esceval.sh shim. Also after some contemplation and testing I have now understood why sed-only approach requires the extra newline: because sed interprets newlines as text delimiters, not as part of the text literal. So getting the closing quote after trailing input newlines only works if you "pad" it with one extra newline. I've edited my answer accordingly as well. (Yes, your "shall" tool is of interest.)
    – mtraceur
    Commented Apr 28, 2015 at 7:55
3

I think this is POSIX. It works by clearing $@ after expanding it for the for loop, but only once so that we can iteratively build it back up (in reverse) using set.

flag=0
for i in "$@"; do
    [ "$flag" -eq 0 ] && shift $#
    set -- "$i" "$@"
    flag=1
done

echo "$@"   # To see that "$@" has indeed been reversed
ls "$@"

I realize reversing the arguments was just an example, but you may be able to use this trick of set -- "$arg" "$@" or set -- "$@" "$arg" in other situations.

And yes, I realize I may have just reimplemented (poorly) ormaaj's Push.

3
  • 1
    Interesting, but probably too specific to the reversing example I happened to choose. More typically I would want to collect multiple argument lists, processing options, etc. Commented Aug 28, 2012 at 15:36
  • To my recent surprise, (( expr )) is not POSIX, although it is widely supported. If you use the construct, all variables are automatically interpolated (no $ needed), which is nice for more complex expressions. Commented Aug 28, 2012 at 16:29
  • I skimmed the spec too quickly; I thought I saw it there. I'll replace with something POSIX, but anyone reading this should feel free to edit my answer with a better test.
    – chepner
    Commented Aug 28, 2012 at 16:36
1

Push. See the readme for examples.

1
  • Seems pretty good, though I was hoping for something compact enough to just be inlined without licensing. Commented Aug 28, 2012 at 15:46
1

The following seems to work with everything I have thrown at it so far, including spaces, both kinds of quotes and a variety of other metacharacters, and embedded newlines:

#!/bin/sh
quote() {
    echo "$1" | sed "s/'/'\"'\"'/g"
}
args=
for arg in "$@"
do
    argq="'"`quote "$arg"`"'"
    args="$argq $args"
done
eval "ls $args"
3
  • This does not handle an argument containing a newline. In fact it is very dangerous because the text in the argument after the newline will be executed as another shell command. Don't run this on your web server!
    – mark4o
    Commented Aug 28, 2012 at 16:09
  • If you give it an argument '-n', it will be turned into '' when quoting, because echo will parse it as an option. Better change echo "$1" into printf "%s" "$1". (dash, Linux) Commented Dec 23, 2015 at 7:20
  • $ eval echo quote "it's" -bash: unexpected EOF while looking for matching `'' -bash: syntax error: unexpected end of file
    – Tom Hale
    Commented Sep 3, 2016 at 7:44
1

If you're okay with calling out to an external executable (as in the sed solutions given in other answers), then you may as well call out to /usr/bin/printf. While it's true that the POSIX shell built-in printf doesn't support %q, the printf binary from Coreutils sure does (since release 8.25).

esceval() {
    /usr/bin/printf '%q ' "$@"
}
0

We can use /usr/bin/printf when version of GNU Coreutil is not less than 8.25

#!/bin/sh

minversion="8.25"
gnuversion=$(ls '--version' | sed '1q' | awk 'NF{print $NF}')

printcmd="printf"

if ! [ $gnuversion \< $minversion ]; then
    printcmd="/usr/bin/printf"
fi;

params=$($printcmd "%q" "$@")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.