It all started as a pretty basic question: Given a char
-- or rather, an integer code point, see Character
API --, return the number of bytes required for its UTF-8 encoding. However, the more time I spent with this innocent little problem, the more confusing it became.
My first approach was:
int getUtf8ByteCount_stdlib(int codePoint) {
int[] codePoints = { codePoint };
String string = new String(codePoints, 0, 1);
byte[] bytes = string.getBytes(StandardCharsets.UTF_8);
return bytes.length;
}
Or for those who like it:
int getUtf8ByteCount_obfuscated(int codePoint) {
return new String(new int[] { codePoint }, 0, 1).getBytes(StandardCharsets.UTF_8).length;
}
Then I created another version (based on UTF-8 wikipedia article) for simplicity and probably efficiency:
int getUtf8ByteCount_handRolled(int codePoint) {
if (codePoint > 0x7FFFFFFF) {
throw new IllegalArgumentException("invalid UTF-8 code point");
}
return codePoint <= 0x7F? 1
: codePoint <= 0x7FF? 2
: codePoint <= 0xFFFF? 3
: codePoint <= 0x1FFFFF? 4
: codePoint <= 0x3FFFFFF? 5
: 6;
}
After years of struggling with the many lovely subtleties of character encoding, I ran a test and lo! it failed; for all code points from '\uD800' to '\uDFFF', the "stdlib" version returns 1 byte versus 3 bytes for "hand-rolled". For sure, it's the good ol' surrogate characters causing havoc again! Now, from my understanding of those pesky little buggers, I would say that the second version is correct. My questions:
- Is
String.getBytes()
or (Java's UTF-8 implementation) broken, or is it my understanding? (I'm using Oracle Java SE Runtime Environment 1.6.0_22-b04) - Even if incorrect, is it preferable to the "hand-rolled" version for being more consistent with the actual byte encoding/decoding produced by Java's UTF-8?
- Correctness considerations apart, do the Java standard libraries provide a cleaner way than my "stlib" one?