52

This is a follow-up to How can I convert JSON-encoded data that contains Unicode surrogate pairs to string?. In that question, the OP had a json.dumps()-encoded file with an emoji represented as a surrogate pair - \ud83d\ude4f. They were having problems reading the file and translating the emoji correctly, and the correct answer was to json.loads() each line from the file, and the json module would handle the conversion from surrogate pair back to (I'm assuming UTF8-encoded) emoji.

So here is my situation: say I have just a regular Python 3 unicode string with a surrogate pair in it:

emoji = "This is \ud83d\ude4f, an emoji."

How do I process this string to get a representation of the emoji out of it? I'm looking to get something like this:

"This is 🙏, an emoji."
# or
"This is \U0001f64f, an emoji."

I've tried:

print(emoji)
print(emoji.encode("utf-8")) # also tried "ascii", "utf-16", and "utf-16-le"
json.loads(emoji) # and `.encode()` with various codecs

Generally I get an error similar to UnicodeEncodeError: XXX codec can't encode character '\ud83d' in position 8: surrogates no allowed.

I'm running Python 3.5.1 on Linux, with $LANG set to en_US.UTF-8. I've run these samples both in the Python interpreter on the command line, and within IPython running in Sublime Text - there don't appear to be any differences.

3

2 Answers 2

65

You've mixed a literal string \ud83d in a json file on disk (six characters: \ u d 8 3 d) and a single character u'\ud83d' (specified using a string literal in Python source code) in memory. It is the difference between len(r'\ud83d') == 6 and len('\ud83d') == 1 on Python 3.

If you see '\ud83d\ude4f' Python string (2 characters) then there is a bug upstream. Normally, you shouldn't get such string. If you get one and you can't fix upstream that generates it; you could fix it using surrogatepass error handler:

>>> "\ud83d\ude4f".encode('utf-16', 'surrogatepass').decode('utf-16')
'🙏'

Python 2 was more permissive.

Note: even if your json file contains literal \ud83d\ude4f (12 characters); you shouldn't get the surrogate pair:

>>> print(ascii(json.loads(r'"\ud83d\ude4f"')))
'\U0001f64f'

Notice: the result is 1 character ( '\U0001f64f'), not the surrogate pair ('\ud83d\ude4f').

0
20

Because this is a recurring question and the error message is slightly obscure, here is a more detailed explanation.

Surrogates are a way to express Unicode code points bigger than U+FFFF.

Recall that Unicode was originally specified to contain 65,536 characters, but that it was soon found that this was not enough to accommodate all the glyphs of the world.

As an extension mechanism for the (otherwise fixed-width) UTF-16 encoding, a reserved area was set up to contain a mechanism for expressing code points outside the Basic Multilingual Plane: Any code point in this special area would have to be followed by another character code from the same area, and together, they would express a code point with a number larger than the old limit.

(Strictly speaking, the surrogates area is divided into two halves; the first surrogate in a pair needs to come from the High Surrogates half, and the second, from the Low Surrogates. Confusingly, the High Surrogates U+D800-U+DBFF have lower code point numbers than the Low Surrogates U+DC00-U+DFFF.)

This is a legacy mechanism to support the UTF-16 encoding specifically, and should not be used in other encodings; they do not need it, and the applicable standards specifically say that this is disallowed.

In other words, while U+12345 can be expressed with the surrogate pair U+D808 U+DF45, you should simply express it directly instead unless you are specifically using UTF-16.

In some more detail, here is how this would be expressed in UTF-8 as a single character:

0xF0 0x92 0x8D 0x85

And here is the corresponding surrogate sequence:

0xED 0xA0 0x88
0xED 0xBD 0x85

As already suggested in the accepted answer, you can round-trip with something like

>>> "\ud808\udf45".encode('utf-16', 'surrogatepass').decode('utf-16').encode('utf-8')
b'\xf0\x92\x8d\x85'

Perhaps see also http://www.russellcottrell.com/greek/utilities/surrogatepaircalculator.htm

9
  • Related:stackoverflow.com/questions/33642339/…
    – tripleee
    Mar 8, 2019 at 12:53
  • this is misleading. "\U0001f64f" is a single character (1 Unicode code point here) in Python (You don't need surrogate pairs to represent it). Now, various character encodings may require several code units to represent it (utf-8 uses 8 bit (1 byte) code units, utf-16 uses 16-bit (2 bytes) code units. Both are variable width encodings: a single unicode code point may require several code units. JSON allows UTF-16 surrogate pairs for \u escapes. But again, it also doesn't require them. In principle, JSON allows to keep the surrogate pair as two code point but Python doesn't support it
    – jfs
    Sep 7, 2022 at 4:18
  • @jfs Which part do you think is misleading? I should perhaps clarify that JSON has no other mechanism than surrogates to express code points outside the BMP.
    – tripleee
    Sep 7, 2022 at 4:29
  • this is wrong: "JSON has no other mechanism than surrogates to express code points outside the BMP" You can put any Unicode character as is inside a json string (except for " quote, and \` slash, or control char ) e.g., "🙏"` is a valid json string according to the standard ecma-international.org/wp-content/uploads/… (you can express it in json via surrogate escapes but you don't have to)
    – jfs
    Sep 7, 2022 at 6:43
  • Right; the correct articulation would be "JSON has no mechanism for specifying a non-BMP code point using a single escape sequence". Some use cases require or at least recommend that you keep your JSON as 7-bit ASCII.
    – tripleee
    Sep 7, 2022 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.