3

I'm a bit new to java, When I assign a unicode string to

  String str = "\u0142o\u017Cy\u0142";
  System.out.println(str);

  final StringBuilder stringBuilder = new StringBuilder();
  InputStream inStream = new FileInputStream("C:/a.txt");
  final InputStreamReader streamReader = new InputStreamReader(inStream, "UTF-8");
  final BufferedReader bufferedReader = new BufferedReader(streamReader);
  String line = "";
  while ((line = bufferedReader.readLine()) != null) {
      System.out.println(line);
      stringBuilder.append(line);
  }

Why are the results different in both cases the file a.txt also contains the same string. but when i print output of the file it prints z\u0142o\u017Cy\u0142 instead of the actual unicode characters. Any idea how do i do this if i want to file content also to be printed as string is being printed.

2
  • does your file literally contains "\u0142o\u017Cy\u0142", with slashes and u's or does it actually contain UTF-8 encoded text?
    – unbeli
    Sep 2, 2010 at 19:55
  • If you are testing this in Eclipse or in Windows it can be tricky to tell if its working or not. Make sure your file is created as a UTF8 text file first and display its contents in a swing text field; that should show you the correct glyphs. Sep 2, 2010 at 19:57

7 Answers 7

5

Your code should be correct, but I guess that the file "a.txt" does not contain the Unicode characters encoded with UTF-8, but the escaped string "\u0142o\u017Cy\u0142".

Please check if the text file is correct, using an UTF-8 aware editor such as recent versions of Notepad or Notepad++ on Windows. Or edit it with your favorite hex editor - it should not contain backslashes.

I tried it with "€" as UTF-8-encoded content of the file and it gets printed correctly. Note that not all Unicode characters can be printed, depending on your terminal encoding (really a hassle on Windows) and font.

5

Java interprets unicode escapes such as your \u0142 that are in the source code as if you had actually typed that character (latin small letter L with stroke) into the source. Java does not interpret unicode escapes that it reads from a file.

If you take your String str = "\u0142o\u017Cy\u0142"; and write it to a file a.txt from your Java program, then open the file in an editor, you'll see the characters themselves in the file, not the \uNNNN sequence.

If you then take your original posted program and read that a.txt file you should see what you expected.

2
  • But how does it work with the property files there i specify the codes in my resource bundle & when i load them & use. They display unicode characters properly
    – Rakesh
    Sep 2, 2010 at 20:17
  • @Rakesh - as BalusC mentioned in his answer, java.util.Properties has a loadConvert() method to do that conversion. My point is that simply reading from a file doesn't do that conversion.
    – Stephen P
    Sep 2, 2010 at 20:47
2

It sounds as though your file literally contains the text z\u0142o\u017Cy\u014, i.e. has Unicode escape sequences in it.

There's probably a library for decoding these but you could do it yourself - according to the Java Language Specification an escape sequence is always of the form \uxxxx, so you could get the 4-digit hex value xxxx for the character, convert it to an integer with Integer.parseInt, convert it to a character and finally replace the whole \uxxxx sequence with the character.

1

So, you want to unescape unicode codepoints? There is no public API available for this. The java.util.Properties has a loadConvert() method which does exactly this, but it's private. Check the Java source for the case you'd like to reuse this. It's doing the conversion by simple parsing. I wouldn't use regex for this since this is too error prone in very specific circumstances.

Or you should probably after all be using java.util.Properties or its i18n counterpart java.util.ResourceBundle with a .properties file instead of a plain .txt file.

See also:

0

I think its just "UTF8" not "UTF-8".

Here I saw it: Source

4
  • 2
    UTF-8 vs UTF8 depends upon whether you use java.io or java.nio. In my experience it doesn't really matter either way. Sep 2, 2010 at 19:55
  • When I gave you said downvote, I was looking at the docs for java.nio.CharSet (used in one of the other constructors for InputStreamReader), which lists it as UTF-8. However, since you've since edited your answer, I was able to cancel said downvote.
    – Powerlord
    Sep 2, 2010 at 20:11
  • UTF-8 is the display name for the encoding; UTF8 is an alias. They are equivalent. Sep 2, 2010 at 20:12
  • @R. Bemrose Ok! @Richard Got it :) Sep 3, 2010 at 6:47
0

I posted Java code to unescape (“descape”?) such things and many others in this answer.

-1

You have used FileInputStream and is a byte code reader not character reader. Try using FileReader instead

something like:

BufferedReader inputStream = new BufferedReader(new FileReader("C:/a.txt"));

then you can use the line oriented I/O BufferedReader to read each line. FileInputREader is a low level I/O that you should avoid. You're writing the characters to your file not the bytes, the best approach is to use character streams. for wrinting and reading unless you need to write bytes/binary data.

2
  • I forgot to mention , try to see your a.txt file in hex and see what you got and you'll understand more from a low level perspective how this things work.
    – Alex
    Sep 3, 2010 at 4:52
  • Say what? He is using a character stream: an InputStreamReader. And character streams have to be built on top of byte streams, so he uses a FileInputStream. The way he's doing it is exactly right. If anything, we should advise people not to use FileReader, because it relies on a platform-dependent system property (i.e., the default encoding).
    – Alan Moore
    Sep 3, 2010 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.