2

I have been working with i18n stuff for a while and thought I knew a lot about this. But I am testing something and the more I look at it, the worse it gets.

We start with a string: Boğaziçi Üniversitesi

It was sent to us by our rich-client application (running Windows and using windows-1254), interpreted as ISO-8859-1 (don't ask) and saved in a MySQL database. Now, in the database, I see the character values below, which can be interpreted as windows-1254 as so:

42 6f f0 61 7a 69 e7 69 20 dc 6e 69 76 65 72 73 69 74 65 73 69
 B  o  ğ  a  z  i  ç  i     Ü  n  i  v  e  r  s  i  t  e  s  i

So far, so good. This looks to be the correct form of the string.

But, here is what one gets by running getBytes() on the string, with either no or different encodings:

BU.getBytes(): (21) 42 6f 3f 61 7a 69 8d 69 20 86 6e 69 76 65 72 73 69 74 65 73 69

BU.getBytes(windows-1254): (21) 42 6f 3f 61 7a 69 e7 69 20 dc 6e 69 76 65 72 73 69 74 65 73 69

BU.getBytes(ISO-8859-1): (21) 42 6f f0 61 7a 69 e7 69 20 dc 6e 69 76 65 72 73 69 74 65 73 69

BU.getBytes(UTF8): (24) 42 6f c3 b0 61 7a 69 c3 a7 69 20 c3 9c 6e 69 76 65 72 73 69 74 65 73 69

So, looking at the last one, one has to wonder where the "ð" came from.

42 6f c3 b0 61 7a 69 c3 a7 69 20 c3 9c 6e 69 76 65 72 73 69 74 65 73 69
B  o      ð  a  z  i     ç  i        Ü  n  i  v  e  r  s  i  t  e  s  i

From http://rishida.net/tools/conversion/, this is what I would expect to get for values in a valid UTF-8 string:

42 6f C4 9F 61 7a 69 C3 A7 69 20 C3 9C 6e 69 76 65 72 73 69 74 65 73 69
 B  o     ğ  a  z  i     ç  i        Ü  n  i  v  e  r  s  i  t  e  s  i

Removing last part of question here and replacing with.

This code:

byte BU_Array[] = new byte[] { (byte)0x42, (byte)0x6F, (byte)0xF0, (byte)0x61,
   (byte)0x7A, (byte)0x69,(byte)0xE7, (byte)0x69, (byte)0x20, (byte)0xDC, 
   (byte)0x6E, (byte)0x69, (byte)0x76, (byte)0x65, (byte)0x72, (byte)0x73, 
    (byte)0x69, (byte)0x74, (byte)0x65, (byte)0x73, (byte)0x69 };

    try {
        String BU_Str_ISO88591 = new String(BU_Array, "ISO-8859-1");
        System.out.println("BU_Str_ISO88591   cP: "+codePointsToHex(BU_Str_ISO88591));

        String BU_Str_W1254 = new String(BU_Array, "windows-1254");
        System.out.println("BU_Str_W1254      cP: "+codePointsToHex(BU_Str_W1254));

        byte bytes_possibly_as_utf8[] = BU_Str_W1254.getBytes("UTF-8");
        System.out.println("bytes from BU_Str_W1254: "+Utilities.bytesToHex(bytes_possibly_as_utf8));

    } catch (java.io.UnsupportedEncodingException uee) {
        uee.printStackTrace();
    }

produces:

BU_Str_ISO88591 cP: (21): 42 6f f0 61 7a 69 e7 69 20 dc 6e 69 76 65 72 73 69 74 65 73 69
BU_Str_W1254 cP: (21): 42 6f 11f 61 7a 69 e7 69 20 dc 6e 69 76 65 72 73 69 74 65 73 69
bytes from BU_Str_W1254: (24)  42 6f c4 9f 61 7a 69 c3 a7 69 20 c3 9c 6e 69 76 65 72 73 69 74 65 73 69

What was confusing here is the 3rd character in the 1st line.

When I took a string that had been given to us as windows-1254, but which we interpreted as iso-8859-1, the code point for the 3rd character is f0. Which is the correct character in windows-1254. Huh? Was that just a coincidence? I doubt it, but the logic seems convoluted.

So, I think I answered my own question here.

3
  • 1
    I see lots of information, but virtually no code. Code is worth 1,024 words! Show the actual code you're using that's performing the conversion that isn't working. Jul 5, 2013 at 21:33
  • 7
    Java Strings are UTF-16 encoded. Only byte[] can be encoded different ways. Jul 5, 2013 at 21:34
  • 2
    To expand on @PeterLawrey's comment, within a Java program there is no choice about String encoding: it's always UTF-16. The only time encoding choices must be made is when converting between String and byte[]. The most common reason for this conversion is input or output, since I/O devices handle data as octet (byte) streams. Jul 5, 2013 at 21:40

0

Browse other questions tagged or ask your own question.