tl;dr
- Use
pgrep
instead of ps
+ grep
- Use
iconv -t UTF8-MAC
to convert your search string to NFD (normalized decomposed Unicode) form.
pgrep -qlf "$(iconv -t UTF8-MAC <<<'amétiq siMed Büro.app')" && echo "RUNNING"
In a nutshell: the Mac filesystem (HFS+) stores filenames in decomposed Unicode form (NFD), whereas what you type into a shell is in composed Unicode form (NFC) and neither the shell nor the Unix utilities treat two equivalent strings - same content, different forms - as content-identical - even though they should.
If the gory details interest you, read on.
Background
Some accented Unicode characters have a composed form - a single code point representing the character directly (e.g. ü
) - as well as an equivalent decomposed form - the base character followed by a combining diacritical character (e.g., u
, followed by ¨
); see https://en.wikipedia.org/wiki/Unicode_equivalence for more information.
Strings that contain only composed characters are in the NFC normal[ized] form (C for 'Composed'), whereas strings that only contain decomposed ones are in the NFD normal[ized] form (D for 'Decomposed').
The Mac filesystem (HFS+) stores filenames in NFD (DEcomposed), which has the following implications:
- Applications launched via Finder and Spotlight are represented as NFD strings in the system's process table.
Similarly, in a shell (bash in Terminal.app), all of the following techniques yield NFD strings:
- pathname expansion (e.g.
echo *.app
)
- output from
ls
and similar utilities
- interactive filename completion at the prompt
By contrast, if you type a script or application name in a shell (or copy a NFC form from elsewhere), it will be represented in NFC.
The crux of the problem: the shell and the Unix utilities do not recognize the equivalence of NFD and NFC forms and therefore treat them as different.
The - cumbersome and obscure - workaround is to only match NFD strings against NFD strings, and only NFC strings against NFC strings.
The insidious thing is that NFD and NFC forms of a given string look absolutely identical in the shell - as they should - but are treated differently.
To determine whether a given string is in NFD or NFC form, use, e.g.:
cat -v <<<'amétiq siMed Büro.app'
- If the string is in NFC, the output is the same as the input.
- If the string is in NFD, if the output contains garbled characters; e.g.,
ame?M-^Atiq siMed Bu?M-^Hro.app
(this, in fact, is what ps
reports - though it shouldn't).
Alternatively, pipe to hexdump -C
to see the individual byte values.
Note that the man
remark about ps
not correctly display argument lists containing multibyte characters is not true per se (at least as of OS X 10.9.2): NFC strings are correctly printed, whereas NFD ones are not.
Contrast that with pgrep
, which prints both NFC and NFD strings correctly, but doesn't recognize their equivalence when matching, as described.
Converting between NFC and NFD forms
- To generically convert any string between NFD and NFC, use
iconv
with the UTF8-MAC
encoding scheme.
The following examples use input string 'ü'
- in NFC form,
$'\xc3\xbc'
- i.e., bytes 0xC3 0xBC
, which is the UTF8 encoding of Unicode codepoint 0xFC
- in NFD form,
$'u\xcc\x88'
- i.e., a u
- the base character - followed by bytes 0xCC 0x88
, which is the UTF8 encoding of Unicode codepoint 0x308
, the so-called combining diaeresis (¨
).
to demonstrate converting; note that in Terminal the result will always appear as ü
- pipe to hexdump -C
, for instance, to see the byte values.
# NFC -> NFD
iconv -t UTF8-MAC <<<$'\xc3\xbc' # -> $'u\xcc\x88'
# NFD -> NFC
iconv -f UTF8-MAC <<<$'u\xcc\x88' # -> $'\xc3\xbc'
These conversions are safe to use in that if the input string is already in the target format, it is left as is.
- To get a reusable ANSI-C-quoted form of a string - whether NFC or NFD - you can use the
bash
shell function quoteNonAscii
listed below; in the case at hand, to get a representation of the application name in NFD form:
cd
to /Applications
(or wherever your application lives)
- Run
quoteNonAscii am*tiq*siMed*B*ro.app
- pathname expansion will ensure that the glob expands to the NFD form of the filename.
# Pass any string to this function to output
# an ANSI-C-quoted string with all non-ASCII bytes represented
# as \x{nn} hex. codes; trailing newlines are always trimmed.
# Examples:
# quoteNonAscii 'ü' # (if NFC) -> $'\xc3\xbc'
# quoteNonAscii 'ü' # (if NFD) -> $'u\xcc\x88'
quoteNonAscii() {
hexdump -ve '/1 "%02x "' <<<"$*" |
awk -v RS=' ' '
BEGIN { printf "$\x27" } # print the opening of the ANSI-C-quoted string, `${single quote}`
$1=="0a" { nls=nls "\x5cn"; next } # store consecutive newlines in a temp. variable
nls { printf "%s", nls; nls="" } # a non-newline char; we now know that the newlines stored so far are NOT trailing, so we print them and clear the temp. variable.
$1>"7f" { printf "\\x" $1; next } # a non-ASCII byte -> PRINT AS `\xnn`
$1=="22" { printf "\x5c\x22"; next } # a double-quote char. -> escape with `\`
$1=="27" { printf "\x5c\x27"; next } # a single-quote char. -> escape with `\`
$1=="07" { printf "\\a"; next } # bell char.
$1=="08" { printf "\\b"; next } # backspace
$1=="09" { printf "\\t"; next } # tab
$1=="0b" { printf "\\v"; next } # vertical tab
$1=="0c" { printf "\\f"; next } # ff
$1=="0d" { printf "\\r"; next } # CR
$1=="1b" { printf "\\e"; next } # escape
{ system("printf %b \"\\x" $1 "\"") } # a byte that is an ASCII char -> print as a CHAR.
END { print "\x27"}' # print the closing `{single quote}` of the ANSI-C-quoted string.
}
Locales in macOS:
Note: This is a revised remnant from the original answer, which hopefully still contains useful information.
- Running
locale
in an interactive shell tells you what locale is in effect, reflected in the following environment variables: LANG
, LC_COLLATE
, LC_CTYPE
, LC_MESSAGES
, LC_MONETARY
, LC_NUMERIC
, LC_TIME
. For instance, if the US English locale is in effect, you'd see:
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
By default, Terminal.app
and other terminal programs such as iTerm
by default preconfigure the locale for shells to match the user's locale as specified via System Preferences > Language & Region
(in Terminal.app
you can turn this behavior off via Preferences... > Settings > {Your Profile} > Advanced
, check box Set locale environment variables on startup
).
The character encoding - reflected in the .{encoding}
suffix in the locale ID, typically .UTF8
- will match the encoding configured in the terminal program's settings (for Terminal.app
, go to Preferences... > Settings > {Your Profile} > Advanced
and change the Character encoding
setting), if supported (use locale -a
to see all supported language/region + encoding combinations).
Both Terminal
and iTerm
default to UTF-8, which is a sensible choice.
If your terminal program is configured to use an unsupported character encoding, the locale ID reported will have NO encoding suffix (e.g., just en_US
) in Terminal
and revert to the "C"
locale altogether in iTerm
- and things will likely NOT work properly (Terminal
will still let you print non-ASCII characters from that encoding, but the utilities won't recognize them as characters, resulting in illegal byte sequence
errors).
- Similarly, if your configure an unsupported combination of primary language and geographic region in
System Preferences
(e.g., combining "German" (de
) with "United States" (US
), which results in supported locale de_US
), only LC_TYPE
will be matched to your terminal program's encoding, and the other LC_*
categories will default to "C"
.
In case you need to set a locale manually, run:
export LANG={localeId}
or
export LC_ALL={localeId}
The difference is that export LANG=...
provides a default for all LC_*
categories while allowing you to selectively override them, whereas export LC_ALL=...
overrides all LC_*
categories.
Supported locale IDs can be listed with locale -a
; it's best to choose one that is UTF-8-based, e.g., de_CH.UTF-8
.
The POSIX locale - essentially an ASCII-only, US-English locale - can be selected either via "POSIX"
or "C"
.
Caveat: ALL Unix utilities that come with macOS suffer the problem described above: they do not recognize equivalent Unicode strings in NFC and NFD as identical.
Aside from this issue, many, but not all Unix utilities are UTF8 multi-byte-character-aware in principle.
- A notable exception as of macOS 10.14 - i.e., a utility that is not UTF8-aware at all - is
awk
; in earlier macOS versions sort
wasn't UTF8-aware either (this changed when the previously used obsolete GNU implementation was replaced with a recent BSD implementation).
locale
output in your bash shell? – mklement0 Apr 22 '14 at 14:09