Is there an easy way to convert a Java string to a true UTF-8 byte array in JNI code?
Unfortunately GetStringUTFChars() almost does what's required but not quite, it returns a "modified" UTF-8 byte sequence. The main difference is that a modified UTF-8 doesn't contain any null characters (so you can treat is an ANSI C null terminated string) but another difference seems to be how Unicode supplementary characters such as emoji are treated.
A character such as U+1F604 "SMILING FACE WITH OPEN MOUTH AND SMILING EYES" is stored as a surrogate pair (two UTF-16 characters U+D83D U+DE04) and has a 4-byte UTF-8 equivalent of F0 9F 98 84, and that is the byte sequence that I get if I convert the string to UTF-8 in Java:
char[] c = Character.toChars(0x1F604);
String s = new String(c);
System.out.println(s);
for (int i=0; i<c.length; ++i)
System.out.println("c["+i+"] = 0x"+Integer.toHexString(c[i]));
byte[] b = s.getBytes("UTF-8");
for (int i=0; i<b.length; ++i)
System.out.println("b["+i+"] = 0x"+Integer.toHexString(b[i] & 0xFF));
The code above prints the following:
😄 c[0] = 0xd83d c[1] = 0xde04 b[0] = 0xf0 b[1] = 0x9f b[2] = 0x98 b[3] = 0x84
However, if I pass 's' into a native JNI method and call GetStringUTFChars() I get 6 bytes. Each of the surrogate pair characters is being converted to a 3-byte sequence independently:
JNIEXPORT void JNICALL Java_EmojiTest_nativeTest(JNIEnv *env, jclass cls, jstring _s)
{
const char* sBytes = env->GetStringUTFChars(_s, NULL);
for (int i=0; sBytes[i]!=0; ++i)
fprintf(stderr, "%d: %02x\n", i, sBytes[i]);
env->ReleaseStringUTFChars(_s, sBytes);
return result;
}
0: ed 1: a0 2: bd 3: ed 4: b8 5: 84
The Wikipedia UTF-8 article suggests that GetStringUTFChars() actually returns CESU-8 rather than UTF-8. That in turn causes my native Mac code to crash because it's not a valid UTF-8 sequence:
CFStringRef str = CFStringCreateWithCString(NULL, path, kCFStringEncodingUTF8);
CFURLRef url = CFURLCreateWithFileSystemPath(NULL, str, kCFURLPOSIXPathStyle, false);
I suppose I could change all my JNI methods to take a byte[] rather than a String and do the UTF-8 conversion in Java but that seems a bit ugly, is there a better solution?