21

Why is the maximum Unicode code point restricted to 0x10FFFF? Is it possible to represent Unicode above this code point - for e.g. 0x10FFFF + 0x000001 = 0x110000 - through any encoding schemes like UTF-16, UTF-8?

0

1 Answer 1

34

It's because of UTF-16. Characters outside of the base multilingual plane (BMP) are represented using a surrogate pair in UTF-16 with the first code unit (CU) lies between 0xD800–0xDBFF and the second one between 0xDC00–0xDFFF. Each of the CU represents 10 bits of the code point, allowing total 20 bits of data (0x100000 characters) which is split into 16 planes (16×216 characters). The remaining BMP will represent 0x10000 characters (code points 0–0xFFFF)

Therefore the total number of characters is 17×216 = 0x100000 + 0x10000 = 0x110000 which allows for code points from 0 to 0x110000 - 1 = 0x10FFFF. Alternatively the last representable code point can be calculated like this: Code points in the BMP are in the range 0–0xFFFF, so the offset for characters encoded with a surrogate pair is 0xFFFF + 1 = 0x10000, which means the last code point that a surrogate pair represents is 0xFFFFF + 0x10000 = 0x10FFFF

That's guaranteed by Unicode Character Encoding Stability Policies that a code point above that will never be assigned

The General_Category property value Surrogate (Cs) is immutable: the set of code points with that value will never change.

Historically UTF-8 allows up to U+7FFFFFFF using 6 bytes whereas UTF-32 can store twice the number of that. However due to the limit in UTF-16 the Unicode committee has decided that UTF-8 can never be longer than 4 bytes, resulting in the same range as UTF-16

In November 2003, UTF-8 was restricted by RFC 3629 to match the constraints of the UTF-16 character encoding: explicitly prohibiting code points corresponding to the high and low surrogate characters removed more than 3% of the three-byte sequences, and ending at U+10FFFF removed more than 48% of the four-byte sequences and all five- and six-byte sequences.

https://en.wikipedia.org/wiki/UTF-8#History

The same has been applied to UTF-32

In November 2003, Unicode was restricted by RFC 3629 to match the constraints of the UTF-16 encoding: explicitly prohibiting code points greater than U+10FFFF (and also the high and low surrogates U+D800 through U+DFFF). This limited subset defines UTF-32

https://en.wikipedia.org/wiki/UTF-32

You can read this more detailed answer and

11
  • to whoever downvoted this: is it too hard to leave a comment if it's wrong?
    – phuclv
    Commented Sep 7, 2018 at 0:31
  • I don’t know who downvoted it but your answer is wrong, but not by a huge margin. The range of BMP is between 0x0000-0xD7FF to 0xE000-0xFFFF. Characters in BMP is be represented in 1 code unit of UTF-16. When 2 code units are used, we have 20 bits to encode a character. Hence this set has 0 to 0xFFFFF values. Since these values in Unicode set should start after BMP, whose last value is 0xFFFF. We add 0x10000 to each character in this set to properly index in the Unicode character set. Hence 0x10000 + 0xFFFFF = 0x10FFFF Commented Oct 26, 2019 at 12:56
  • I voted down this question, because it doesn't answer the question. WHY would unicode ever create something as backward thinking as UTF-16. They were supposed to overcome the problem of fixed size code-point ranges. What happens when you humans make first contact and are expected to add support to the 1500000+ new scripts from the 10000+ new civilizations, are you just going to say "no, sorry, that is not UTF-16 compatible"; this could prompt discussions about the appropriateness of the future of human existence on Earth. Commented Jul 6, 2023 at 18:50
  • @user3338098 It already answers the OP's question. It's because of UTF-16. And WHY would unicode ever create something as backward thinking as UTF-16 is your question, not the OP's and that's a completely different one. Downvoting this because it doesn't answer your question is silly. If you want then ask the other question. The oversight of Unicode committee has nothing to do with this
    – phuclv
    Commented Jul 7, 2023 at 3:33
  • @phuclv "Why sky blue? Because sky has air! [enlightenment achieved]", while some may be satisfied with an answer of "sky has air", I am not one of them. However an answer of "Because sky has air, but I do not think anyone knows why air blue." would be an acceptable answer, if a bit hard to believe. Commented Jul 7, 2023 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.