87

I was playing around with Unicode identifiers and stumbled upon this:

>>> ๐‘“, x = 1, 2
>>> ๐‘“, x
(1, 2)
>>> ๐‘“, f = 1, 2
>>> ๐‘“, f
(2, 2)

What's going on here? Why does Python replace the object referenced by ๐‘“, but only sometimes? Where is that behavior described?

5
  • 10
    This is an interesting question, but your minimal reproducible example could have just been ๐‘“=1 f=2 print(๐‘“)
    – khelwood
    Jun 8, 2020 at 6:18
  • 1
    Thanks. Made the example even smaller now. Jun 8, 2020 at 6:31
  • 37
    obxkcd
    – Barmar
    Jun 8, 2020 at 14:30
  • 1
    a, a = 1, 2; a, a. This has nothing to do with f or ๐‘“.
    – user76284
    Jun 9, 2020 at 3:13
  • 4
    The example ๐‘“ = 3; f would suffice.
    – user76284
    Jun 9, 2020 at 3:15

2 Answers 2

87

PEP 3131 -- Supporting Non-ASCII Identifiers says

All identifiers are converted into the normal form NFKC while parsing; comparison of identifiers is based on NFKC.

You can use unicodedata to test the conversions:

import unicodedata

unicodedata.normalize('NFKC', '๐‘“')
# f

which would indicate that '๐‘“' gets converted to 'f' in parsing. Leading to the expected:

๐‘“  = "Some String"
print(f)
# "Some String"
8
  • 26
    This is a great answer, but a terrible decision by the Python core devs. I note that in the discussion of this PEP, one of the objections was that Unicode is poorly-understood and has weak tooling. Now, over a decade later, I wonder if it's time to re-think the romanization of Unicode identifiers.
    – Adam Smith
    Jun 8, 2020 at 6:48
  • 36
    @AdamSmith but Unicode normalisation isn't romanisation. You can have ฯ€ as a Python identifier that is distinct from p just fine. If I understand correctly, the NFK* folding is about characters that the Unicode folks thought should have been the same character to begin with, but they can't be merged because of backwards-compatibility with some legacy encodings.
    – lenz
    Jun 8, 2020 at 8:33
  • 21
    There are two kinds of character equivalence: canonical and compatibility. Canonical equivalence should render the exact same glyph, which is not the case between ๐‘“ and f. NFKC normalizes both canonical and compatibility equivalences, which I agree is a bad choice for a programming language like Python, who differentiates between letter cases: it is expected that identifiers that render differently should be different. Python should have used NFC, which ensures ๐‘“ and f are different things.
    – lvella
    Jun 8, 2020 at 14:52
  • 30
    Some form of normalization is needed because of, for example, latin characters with diacritics - if I see a character like 'ü' then it might be either a composite character (u + combining diaeresis) or a precomposed single character; the user would have no reasonable way or desire to distinguish them, and their preferred input method would likely allow to input only one of these options. So it's desirable that if i see 'ü' and type 'ü' then the language considers the characters as equivalent even if they're encoded differently, though NFC normalization would probably be sufficient for that.
    – Peteris
    Jun 8, 2020 at 16:15
  • 10
    Python supports Unicode for identifiers in order to facilitate its use in defining identifiers in non-English languages, not to provide equal access to all Unicode code points. For example, it is currently quite difficult to hack the parser to support Unicode operators, because any non-ASCII character is first assumed to be part of an identifier, even if the Unicode character in question isn't a valid part of an identifier. The idea is not to support mining Unicode for "interesting" characters, but to support characters produced by standard non-English keyboard layouts.
    – chepner
    Jun 8, 2020 at 19:24
31

Here's a small example, just to show how horrible this "feature" is:

๐•‹๐กแตข๐”ฐ_๏ฝ†๐”ข๐˜ข๐š๐“Šแตฃโ‚‘_๐•คโ‚•๐”ฌ๐”ฒ๐–‘๐”ก_dโ‚‘๐•—แตข๐˜ฏ๏ฝ‰๐˜ต๐šŽโ„“y_๐’ท๐˜ฆ_๐š_๐š‹แต˜g = 42
print(T๐—ตโ„น๐šœ_๐’‡e๐–†๐š๐™ชแตฃe_โ‚›๐”ฅยบ๐“พ๐—น๐™™_๐še๐’‡แตข๐’โฑ๏ฝ”แต‰๐•๐˜†_๐–ปโ„ฏ_๐”ž_๐–‡๐–š๐“ฐ)
# => 42

Try it online! (But please don't use it)

And as mentioned by @MarkMeyer, two identifiers might be distinct even though they look just the same ("CYRILLIC CAPITAL LETTER A" and "LATIN CAPITAL LETTER A")

ะ = 42
print(A)
# => NameError: name 'A' is not defined
10
  • 3
    Makes me want to write an equivalent of jsfuck.com... python-unicode-hell.com ? Jun 8, 2020 at 17:28
  • 2
    @MathieuVIALES ๐“•๐•–๐’†๐‘™ ๐Ÿสณ๐™šโ‚‘ แต—๐—ˆ แตˆ๐š˜ ๐“ˆº. I ๐ก๏ฝ๐”ณแต‰ ๐’”๐š˜๐™ข๐–พ ๐’„๐‘œ๐–ฝแต‰ ๐–‘๐’ถ๐’š๐‘–๐’๐•˜ ๏ฝrโ‚’๐˜ถ๐˜ฏ๐–ฝ. ๐ˆ สท๐™–n๐“‰โ„ฏ๐™™ ๐’•๐˜ฐ ๐—๐•ฃ๏ฝ๐‘™๐—… โ…ฝ๐”ฌ๐š•๐˜ญแต‰๐—ฎ๐“ฐ๐˜ถ๐–Š๐”ฐ สท๐š’โ‚œ๐™ ๐“ฒแต—, ๐•“๐’–๏ฝ” ๐šโ„Žโ‚‘ ๐—‹๐‘’๐™จ๐“Š๐•๐“‰ โ…ˆ๐”ฐ ๐“ณแต˜๐‘ ๐™ฉ ๏ฝ”๐š˜๐—ˆ ๐—ต๏ฝ๐’“๐‘Ÿibl๏ฝ… ๐˜€๐จ ๐ผ โฟ๐šŽ๏ฝ–๐–พ๐”ฏ แต˜๐“ผโ‚‘โ…พ โ…ˆt. ๐•Œ๐“ƒ๐—๐š’๏ฝŒ ๐•Ÿ๐š˜๐™ฌ. Jun 8, 2020 at 17:32
  • 8
    And then of course: ะ = 42; print(A) --> "NameError: name 'A' is not defined"
    – Mark
    Jun 8, 2020 at 17:54
  • 10
    The point was never to open the door to arbitrarily complex identifier names, but to facilitate typing identifiers in a programmer's native language (using a keyboard layout native to that language). Better to go by Unicode's classification of a code point as a letter than to act as the arbiter for which writing systems can and cannot be used for identifiers. (And limiting an identifier to characters from a single writing system is far beyond the parser's pay grade.)
    – chepner
    Jun 8, 2020 at 19:32
  • 13
    None of those code points are part of any natural language's writing system, so whether any of them are acceptable as part of an identifier is almost "accidental", based on Unicode classification rather than any explicit endorsement by Python itself.
    – chepner
    Jun 8, 2020 at 20:00

Your Answer

By clicking โ€œPost Your Answerโ€, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.