79

Recently I have been having some trouble using GnuWin32 from PowerShell whenever double quotes are involved.

Upon further investigation, it appears PowerShell is stripping double quotes from command line arguments, even when properly escaped.

PS C:\Documents and Settings\Nick> echo '"hello"'
"hello"
PS C:\Documents and Settings\Nick> echo.exe '"hello"'
hello
PS C:\Documents and Settings\Nick> echo.exe '\"hello\"'
"hello"

Notice that the double quotes are there when passed to PowerShell's echo cmdlet, but when passed as an argument to echo.exe, the double quotes are stripped unless escaped with a backslash (even though PowerShell's escape character is a backtick, not a backslash).

This seems like a bug to me. If I am passing the correct escaped strings to PowerShell, then PowerShell should take care of whatever escaping may be necessary for however it invokes the command.

What is going on here?

For now, the fix is to escape command line arguments in accordance with these rules (which seem to be used (indirectly) by the CreateProcess API call which PowerShell uses to invoke .exe files):

  • To pass a double quote, escape with a backslash: \" -> "
  • To pass a one or more backslashes followed by a double quote, escape each backslash with another backslash and escape the quote: \\\\\" -> \\"
  • If not followed by a double quote, no escaping is necessary for backslashes: \\ -> \\

Note that further escaping of double quotes may be necessary to escape the double quotes in the Windows API escaped string to PowerShell.

Here are some examples, with echo.exe from GnuWin32:

PS C:\Documents and Settings\Nick> echo.exe "\`""
"
PS C:\Documents and Settings\Nick> echo.exe "\\\\\`""
\\"
PS C:\Documents and Settings\Nick> echo.exe "\\"
\\

I imagine that this can quickly become hell if you need to pass a complicated command line parameter. Of course, none of this documented in the CreateProcess() or PowerShell documentation.

Also note that this is not necessary to pass arguments with double quotes to .NET functions or PowerShell cmdlets. For that, you need only escape your double quotes to PowerShell.

Edit: As Martin pointed out in his excellent answer, this is documented in the CommandLineToArgv() function (which the CRT uses to parse the command line arguments) documentation.

9
  • what version of powershell are you using? v1 was a hell of a lot worse with quotes. v2 is much better but still suffers. v2 ships with win7+2008r2 but is installable on 2003/xp too.
    – x0n
    Jul 17, 2011 at 2:59
  • 1
    I consider it a bug, but PS is not stripping the quotes. As you found out in your last example, PS is simply failing to escape quotes when executing the application. Jul 18, 2011 at 21:22
  • 1
    I have found that most of my problems have been from trying too hard. Most of the time PowerShell quoting works as expected. A handy tool is showargs.exe that can display the command-line that PowerShell is actually running. Mar 6, 2015 at 22:32
  • Note that echo is an alias for Write-Output.
    – jpmc26
    Feb 3, 2018 at 3:51
  • 1
    Here we are 8 years later and my current fix is just not using powershell when I have to pass double-quotes to some program.
    – nw.
    Sep 13, 2019 at 0:18

7 Answers 7

56

It is a known thing:

It's FAR TOO HARD to pass parameters to applications which require quoted strings. I asked this question in IRC with a "roomful" of PowerShell experts, and it took hour for someone to figure out a way (I originally started to post here that it is simply not possible). This completely breaks PowerShell's ability to serve as a general purpose shell, because we can't do simple things like executing sqlcmd. The number one job of a command shell should be running command-line applications... As an example, trying to use SqlCmd from SQL Server 2008, there is a -v parameter which takes a series of name:value parameters. If the value has spaces in it, you must quote it...

...there is no single way to write a command line to invoke this application correctly, so even after you master all 4 or 5 different ways of quoting and escaping things, you're still guessing as to which will work when ... or, you can just shell out to cmd, and be done with it.

6
  • 9
    Wow, shocking that this has not been fixed after a year and a half.
    – nw.
    Jul 16, 2011 at 0:06
  • 9
    It says 'closed as fixed' but there's no link to the fix or how to use it. This sucks. Jun 6, 2012 at 15:10
  • 3
    The link is (effectively) broken - "Microsoft Connect Has Been Retired" Mar 5, 2019 at 21:18
  • 1
    Today I experienced another bad thing about ": when using CMD /C command where one needs to add " to filenames in the command because they contain spaces, in Win7 you need extra " before and after the command, in Win10 it is an error. Had to resolve it by finding the system version: # system version $ver = [Environment]::OSVersion.Version.ToString(); $ver = $ver.Substring(0,2); # in windows 7 add " before and after If ($ver -Match "10") {$cc = $c} Else {$cc = '"' + $c + '"'} cmd /c $cc
    – Martin
    Mar 29, 2019 at 19:24
  • 4
    Ohhh I see. All people who attempted to fix this have retired and no longer work in technology industry having failed to solve this problem
    – KCD
    Apr 29, 2019 at 0:22
53

TL;DR

If you just want a solution for Powershell 5, see:

ConvertTo-ArgvQuoteForPoSh.ps: Powershell V5 (and C# Code) to allow escaping native command arguments

The Question I will try to answer

..., it appears PowerShell is stripping double quotes from command line arguments, even when properly escaped.

PS C:\Documents and Settings\Nick> echo.exe '"hello"'
hello 
PS C:\Documents and Settings\Nick> echo.exe '\"hello\"' 
"hello"

Notice that the double quotes are there when passed to PowerShell's echo cmdlet, but when passed as an argument to echo.exe, the double quotes are stripped unless escaped with a backslash (even though PowerShell's escape character is a backtick, not a backslash).

This seems like a bug to me. If I am passing the correct escaped strings to PowerShell, then PowerShell should take care of whatever escaping may be necessary for however it invokes the command.

What is going on here?

The Non-Powershell Background

The fact that you need to escape the quotes with backslashes \ has nothing to to with powershell, but with the CommandLineToArgvW function that is used by all msvcrt and C# programs to build the argv array from the single-string command line that the Windows process gets passed.

The details are explained at Everyone quotes command line arguments the wrong way and it basically boils down to the fact that this function historically has very uninutitive escaping rules:

  • 2n backslashes followed by a quotation mark produce n backslashes followed by begin/end quote. This does not become part of the parsed argument, but toggles the "in quotes" mode.
  • (2n) + 1 backslashes followed by a quotation mark again produce n backslashes followed by a quotation mark literal ("). This does not toggle the "in quotes" mode.
  • n backslashes not followed by a quotation mark simply produce n backslashes.

leading to the described generic escaping function (shortquote of the logic here):

CommandLine.push_back (L'"');

for (auto It = Argument.begin () ; ; ++It) {
      unsigned NumberBackslashes = 0;

      while (It != Argument.end () && *It == L'\\') {
              ++It;
              ++NumberBackslashes;
      }

      if (It == Argument.end ()) {
              // Escape all backslashes, but let the terminating
              // double quotation mark we add below be interpreted
              // as a metacharacter.
              CommandLine.append (NumberBackslashes * 2, L'\\');
              break;
      } else if (*It == L'"') {
              // Escape all backslashes and the following
              // double quotation mark.
              CommandLine.append (NumberBackslashes * 2 + 1, L'\\');
              CommandLine.push_back (*It);
      } else {
              // Backslashes aren't special here.
              CommandLine.append (NumberBackslashes, L'\\');
              CommandLine.push_back (*It);
      }
}

CommandLine.push_back (L'"');

The Powershell specifics

Now, up to Powershell 5 (including PoSh 5.1.18362.145 on Win10/1909) PoSh knows basically diddly about these rules, nor should it arguably, because these rules are not really general, because any executable you call could, in theory, use some other means to interpret the passed command line.

Which leads us to -

The Powershell Quoting Rules

What PoSh does do however is try to figure out whether the strings you pass it as arguments to the native commands need to be quoted because they contain whitespace.

PoSh - in contrast to cmd.exe - does a lot more parsing on the command you hand it, since it has to resolve variables and knows about multiple arguments.

So, given a command like

$firs  = 'whaddyaknow'
$secnd = 'it may have spaces'
$third = 'it may also have "quotes" and other \" weird \\ stuff'
EchoArgs.exe $firs $secnd $third

Powershell has to take a stance on how to create the single string CommandLine for the Win32 CreateProcess (or rather the C# Process.Start) call it will evetually have to do.

The approach Powershell takes is weird and got more complicated in PoSh V7 , and as far as I can follow, it's got to do how powershell treats unbalanced quotes in unquoted string. The long stories short is this:

Powershell will auto-quote (enclose in <">) a single argument string, if it contains spaces and the spaces don't mix with an uneven number of (unsescaped) double quotes.

The specific quoting rules of PoSh V5 make it impossible to pass a certain category of string as single argument to a child process.

PoSh V7 fixed this, so that as long as all quotes are \" escaped -- which they need to be anyway to get them through CommandLineToArgvW -- we can pass any aribtrary string from PoSh to a child executable that uses CommandLineToArgvW.

Here's the rules as C# code as extracted from the PoSh github repo for a tool class of ours:

PoSh Quoting Rules V5

    public static bool NeedQuotesPoshV5(string arg)
    {
        // bool needQuotes = false;
        int quoteCount = 0;
        for (int i = 0; i < arg.Length; i++)
        {
            if (arg[i] == '"')
            {
                quoteCount += 1;
            }
            else if (char.IsWhiteSpace(arg[i]) && (quoteCount % 2 == 0))
            {
                // needQuotes = true;
                return true;
            }
        }
        return false;
    }

PoSh Quoting Rules V7

    internal static bool NeedQuotesPoshV7(string arg)
    {
        bool followingBackslash = false;
        // bool needQuotes = false;
        int quoteCount = 0;
        for (int i = 0; i < arg.Length; i++)
        {
            if (arg[i] == '"' && !followingBackslash)
            {
                quoteCount += 1;
            }
            else if (char.IsWhiteSpace(arg[i]) && (quoteCount % 2 == 0))
            {
                // needQuotes = true;
                return true;
            }

            followingBackslash = arg[i] == '\\';
        }
        // return needQuotes;
        return false;
    }

Oh yeah, and they also added in a half baked attempt to correctly escape the and of the quoted string in V7:

if (NeedQuotes(arg))
{
      _arguments.Append('"');
      // need to escape all trailing backslashes so the native command receives it correctly
      // according to http://www.daviddeley.com/autohotkey/parameters/parameters.htm#WINCRULESDOC
      _arguments.Append(arg);
      for (int i = arg.Length - 1; i >= 0 && arg[i] == '\\'; i--)
      {
              _arguments.Append('\\');
      }

      _arguments.Append('"');

The Powershell Situation

Input to EchoArgs             | Output V5 (powershell.exe)  | Output V7 (pwsh.exe)
===================================================================================
EchoArgs.exe 'abc def'        | Arg 0 is <abc def>          | Arg 0 is <abc def>
------------------------------|-----------------------------|---------------------------
EchoArgs.exe '\"nospace\"'    | Arg 0 is <"nospace">        | Arg 0 is <"nospace">
------------------------------|-----------------------------|---------------------------
EchoArgs.exe '"\"nospace\""'  | Arg 0 is <"nospace">        | Arg 0 is <"nospace">
------------------------------|-----------------------------|---------------------------
EchoArgs.exe 'a\"bc def'      | Arg 0 is <a"bc>             | Arg 0 is <a"bc def>
                              | Arg 1 is <def>              |
------------------------------|-----------------------------|---------------------------
   ...

I'm snipping further examples here for time reasons. They shouldn't add overmuch to the answer anyways.

The Powershell Solution

To pass arbitrary Strings from Powershell to a native command using CommandLineToArgvW, we have to:

  • properly escape all quotes and Backslashes in the source argument
    • This means recognizing the special string-end handling for backslashes that V7 has. (This part is not implemented in the code below.)
  • and determine whether powershell will auto-quote our escaped string and if it won't auto-quote it, quote it ourselves.
    • and make sure that the string we quoted ourselves then doesn't get auto-quoted by powershell: This is what breaks V5.

Powershell V5 Source code for correctly escaping all arguments to any native command

I've put the full code on Gist, as it got too long to include here: ConvertTo-ArgvQuoteForPoSh.ps: Powershell V5 (and C# Code) to allow escaping native command arguments

  • Note that this code tries it's best, but for some strings with quotes in the payload and V5 you simply must add in leading space to the arguments you pass. (See code for logic details).
5
  • 4
    Thank you for the comprehensive explanation. This is horrible.
    – bb010g
    Sep 16, 2020 at 9:11
  • So, in other words, when compiling a c source with gcc and trying to define a quoted macro with -D, this is impossible? Aug 1, 2021 at 18:15
  • @RenéNyffenegger - why on earth would you call gcc from PoSh? ;-) Quite frankly, I have no clue ... I guess try it and see?
    – Martin Ba
    Aug 17, 2021 at 20:16
  • 1
    @MartinBa Because I want to compile a C source into an executable or a DLL. I tried quite hard to define a quoted macro, but didn't succeed, which is why I eventually stumbled upon this question. Aug 18, 2021 at 3:52
  • 1
    "PoSh knows basically diddly about these rules, nor should it arguably" - unless a concerted effort is made by MS to correct the problem, then it arguably should. Also, regarding Rene's question, it's pretty obvious that if PS wants to be a cmd.exe replacement it needs to let you call stuff - gcc is a pretty common call from a command line, so your response is quite unhelpful. Kudos for writing code and posting it though.
    – Derek
    Nov 3, 2022 at 4:32
17

I personally avoid using '\' to escape things in PowerShell, because it's not technically a shell escape character. I've gotten unpredictable results with it. In double-quoted strings, you can use "" to get an embedded double-quote, or escape it with a back-tick:

PS C:\Users\Droj> "string ""with`" quotes"
string "with" quotes

The same goes for single quotes:

PS C:\Users\Droj> 'string ''with'' quotes'
string 'with' quotes

The weird thing about sending parameters to external programs is that there is additional level of quote evaluation. I don't know if this is a bug, but I'm guessing it won't be changed, because the behavior is the same when you use Start-Process and pass in arguments. Start-Process takes an array for the arguments, which makes things a bit clearer, in terms of how many arguments are actually being sent, but those arguments seem to be evaluated an extra time.

So, if I have an array, I can set the argument values to have embedded quotes:

PS C:\cygwin\home\Droj> $aa = 'arg="foo"', 'arg=""""bar""""'
PS C:\cygwin\home\Droj> echo $aa
arg="foo"
arg=""""bar""""

The 'bar' argument has enough to cover the extra hidden evaluation. It's as if I send that value to a cmdlet in double-quotes, then send that result again in double-quotes:

PS C:\cygwin\home\Droj> echo "arg=""""bar""""" # level one
arg=""bar""
PS C:\cygwin\home\Droj> echo "arg=""bar""" # hidden level
arg="bar"

One would expect these arguments to be passed to external commands as-is, as they are to cmdlets like 'echo'/'write-output', but they are not, because of that hidden level:

PS C:\cygwin\home\Droj> $aa = 'arg="foo"', 'arg=""""bar""""'
PS C:\cygwin\home\Droj> start c:\cygwin\bin\echo $aa -nonew -wait
arg=foo arg="bar"

I don't know the exact reason for it, but the behavior is as if there is another, undocumented step being taken under the covers that re-parses the strings. For example, I get the same result if I send the array to a cmdlet, but add a parsing level by doing it through invoke-expression:

PS C:\cygwin\home\Droj> $aa = 'arg="foo"', 'arg=""""bar""""'
PS C:\cygwin\home\Droj> iex "echo $aa"
arg=foo
arg="bar"

...which is exactly what I get when I send these arguments to my external Cygwin instance's 'echo.exe':

PS C:\cygwin\home\Droj> c:\cygwin\bin\echo 'arg="foo"' 'arg=""""bar""""'
arg=foo arg="bar"
1
  • 1
    This approach is nice because it also allows you to expand PowerShell variables in the string, e.g.: "name=`"$value`"" will result in name="<value>" Aug 15, 2013 at 9:39
5

With PowerShell 7.2.0, it is finally possible for arguments passed to native executables to behave as expected. This is currently an experimental feature and needs to be enabled manually.

Enable-ExperimentalFeature PSNativeCommandArgumentPassing

After that edit your PSProfile, for example, using notepad:

notepad.exe $PROFILE

Add $PSNativeCommandArgumentPassing = 'Standard' to the top of the file. You may instead also use $PSNativeCommandArgumentPassing = 'Windows' which uses the Legacy behaviour for some native executables. The differences are documented in this pull request.

Finally, restart PowerShell. Command arguments will no longer have quotes removed.


The new behaviour can be verified with this little C program:

#include <stdio.h>

int main(int argc, char** argv) {
    for (int i = 1; i < argc; i++) {
        puts(argv[i]);
    }
    return 0;
}

Compile it with gcc and pass in some arguments with quotes, like a JSON string.

> gcc echo-test.c
> ./a.exe '{"foo": "bar"}'

With the Legacy behaviour, the output is {foo: bar}. However, with the Standard option, the output becomes {"foo": "bar"}.

1
  • Fantastic! Finally!
    – nw.
    Mar 17, 2022 at 22:05
2

Relying on the CMD to shell out the issue as indicated in the accepted answer didn't work for me as double quotes were still stripped out when calling the CMD executable.

The good solution for me was to structure my command line as an array of strings instead of a single full string containing all the arguments. Then simply pass that array as the arguments for the binary invocation:

$args = New-Object System.Collections.ArrayList
$args.Add("-U") | Out-Null
$args.Add($cred.UserName) | Out-Null
$args.Add("-P") | Out-Null
$args.Add("""$($cred.Password)""")
$args.Add("-i") | Out-Null
$args.Add("""$SqlScriptPath""") | Out-Null
& SQLCMD $args

In that case, double quotes surrounding arguments are properly passed to the invoked command.

If you need, you can test and debug it with EchoArgs from the PowerShell Community Extensions.

1
  • This isn't completely PowerShell's fault. sqlcmd has some really weird behavior, too. Try this command in Command Prompt: sqlcmd Q "SELECT '$(x)'" -v x= """hello world""". I get ""hello world"" inside, when I should get "hello world". It's like sqlcmd bypasses the normal escaping rules somehow, too. That said, you can definitely pass arguments that contain spaces just by passing a normal PowerShell string: sqlcmd -Q "SELECT '`$(x)'" -v x= 'hello world' gives me hello world.
    – jpmc26
    Feb 3, 2018 at 5:55
0

Oh dear. Clearly trying to escape double quotes to get them into PowerShell from the command line, or worse, some other language you are using to generate such a command line, or execution environments which might chain PowerShell scripts, can be a colossal waste of time.

As an attempt at a practical solution, what can we do instead? Silly-looking workarounds can sometimes be effective:

powershell Write-Host "'say ___hi___'.Replace('___', [String][Char]34)"

But it depends a lot on how this is being executed. Note that if you want that command to have the same results when pasted in PowerShell instead of run from command prompt, you need those outer double quotes! Because the hosting Powershell turns the expression into a string object which becomes just one more parameter to 'powershell.exe'

PS> powershell Write-Host 'say ___hi___'.Replace('___', [String][Char]34)

Which then, I guess, parses its arguments as Write-Host say "hi"

So the quotes you are trying so hard to reintroduce with string.Replace() will just disappear!

3
0

This seems to be fixed in recent versions of PowerShell at the time of this writing, so it is no longer something to worry about.

If you still think you see this issue, remember it may be related to something else, such as the program that invokes PowerShell, so if you cannot reproduce it when invoking PowerShell directly from a command prompt or the ISE, you should debug elsewhere.

For example, I found this question when investigating a problem of disappearing quotes when running a PowerShell script from C# code using Process.Start. The issue was actually C# Process Start needs Arguments with double quotes - they disappear.

7
  • Still no luck in my case with using this for SqlCmd and -v option which requires the double quote escaping on variable values containing spaces
    – GGirard
    Dec 19, 2017 at 15:50
  • 1
    I understand that my answer isn't particularly helpful. I wrote it because I spent hours looking for the problem in the way PowerShell handles quotes when it was actually somewhere else, and others may be in the same situation. My hope is that my answer has helped or will help people by making them look for the problem elsewhere and debugging their script by running different commands, etc. Dec 20, 2017 at 10:56
  • 6
    I think this answer is factually incorrect, because there is no evidence that this has been fixed. Try everything in OP in the latest powershell and you will get exactly the same results. Jan 29, 2019 at 3:38
  • pwsh 6.1.2 and the quotes still disappear no quote is shown when calling EchoArgs.exe "quote" '""twoquotes""' """""4quotes"""""
    – stej
    Jan 29, 2019 at 8:42
  • 1
    I'm not sure what "no finite complete list" has to do with that. I repeat myself: Try everything in the OP, this is very finite and not very long list. Mar 28, 2019 at 3:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.