138

I tried to use java.io.FileReader to read some text files and convert them into a string, but I found the result is wrongly encoded and not readable at all.

Here's my environment:

  • Windows 2003, OS encoding: CP1252

  • Java 5.0

My files are UTF-8 encoded or CP1252 encoded, and some of them (UTF-8 encoded files) may contain Chinese (non-Latin) characters.

I use the following code to do my work:

   private static String readFileAsString(String filePath)
    throws java.io.IOException{
        StringBuffer fileData = new StringBuffer(1000);
        FileReader reader = new FileReader(filePath);
        //System.out.println(reader.getEncoding());
        BufferedReader reader = new BufferedReader(reader);
        char[] buf = new char[1024];
        int numRead=0;
        while((numRead=reader.read(buf)) != -1){
            String readData = String.valueOf(buf, 0, numRead);
            fileData.append(readData);
            buf = new char[1024];
        }
        reader.close();
        return fileData.toString();
    }

The above code doesn't work. I found the FileReader's encoding is CP1252 even if the text is UTF-8 encoded. But the JavaDoc of java.io.FileReader says that:

The constructors of this class assume that the default character encoding and the default byte-buffer size are appropriate.

Does this mean that I am not required to set character encoding by myself if I am using FileReader? But I did get wrongly encoded data currently, what's the correct way to deal with my situtaion? Thanks.

4
  • You should also loose the String.valueOf() inside the loop and use StringBuffer.append(char[],int,int) directly. This saves a lot of copying of the char[]. Also replace StringBuffer with StringBuilder. None of this is about your question, 'though. Mar 30, 2009 at 12:01
  • 1
    I hate to say it, but have you read the JavaDoc right after the part you pasted? You know, the part that says "To specify these values yourself, construct an InputStreamReader on a FileInputStream."?
    – Powerlord
    Mar 30, 2009 at 13:55
  • Thanks for your comment, actually I read the JavaDoc, but what I am not sure is whether or not I should specify these values myself, and switch to "construct an InputStreamReader on a FileInputStream".
    – nybon
    Mar 31, 2009 at 1:05
  • Yes, if you know the file is in something other than the platform default encoding, you have to tell the InputStreamReader which one to use.
    – Alan Moore
    Mar 31, 2009 at 4:46

6 Answers 6

271

Yes, you need to specify the encoding of the file you want to read.

Yes, this means that you have to know the encoding of the file you want to read.

No, there is no general way to guess the encoding of any given "plain text" file.

The one-arguments constructors of FileReader always use the platform default encoding which is generally a bad idea.

Since Java 11 FileReader has also gained constructors that accept an encoding: new FileReader(file, charset) and new FileReader(fileName, charset).

In earlier versions of java, you need to use new InputStreamReader(new FileInputStream(pathToFile), <encoding>).

10
  • 1
    InputStream is = new FileInputStream(filename); here i got error file not found error with Russian file name Feb 10, 2014 at 8:59
  • 3
    +1 for the suggestion of using InputStreamReader, however using links in code blocks makes it hard to copy and paste the code, if this can be changed, thx
    – Ferrybig
    Sep 26, 2015 at 16:12
  • 1
    Would it be "UTF-8" or "UTF8" in the encodings. According to the Java SE reference on encoding, since InputStreamReader is a java.io class, it would be "UTF8"? Nov 13, 2015 at 19:01
  • 12
    @NobleUplift: the safest bet is StandardCharsets.UTF_8, there's no chance of mistyping there ;-) But yes, if you go with string "UTF8" would be correct (although I seem to remember that it will accept both ways). Nov 13, 2015 at 19:53
  • 1
    @JoachimSauer Actually, this is one of the purposes of the Byte Order Mark, along with.. well.. establishing the byte order! :) As such I find it weird that Java's FileReader is not able to automatically detect UTF-16 that has such a BOM... In fact I once wrote a UnicodeFileReader that does exactly that. Unfortunately closed source, but Google has it's UnicodeReader which is very similar. Nov 20, 2015 at 16:02
80

FileReader uses Java's platform default encoding, which depends on the system settings of the computer it's running on and is generally the most popular encoding among users in that locale.

If this "best guess" is not correct then you have to specify the encoding explicitly. Unfortunately, FileReader does not allow this (major oversight in the API). Instead, you have to use new InputStreamReader(new FileInputStream(filePath), encoding) and ideally get the encoding from metadata about the file.

8
  • 24
    "major oversight in the API" - thanks for this explanation - I was wondering why I couldn't find the constructor I was after ! Cheers John
    – monojohnny
    Apr 12, 2013 at 13:42
  • @Bhanu Sharma: that's an encoding issue at a different level, check where you're getting the filename from, and if it's hardcoded what encoding the compiler uses. Feb 10, 2014 at 9:27
  • 1
    @BhanuSharma: filename encoding issues are nothing to do with this question. See one of the many existing “why don't Unicode filenames work in Java” questions. Spoiler: java.io APIs like FileReader use C standard library filesystem calls, which can't support Unicode on Windows; consider using java.nio instead.
    – bobince
    May 12, 2015 at 7:48
  • 1
    "FileReader uses Java's platform default encoding, which depends on the system settings of the computer it's running on and is generally the most popular encoding among users in that locale." I wouldn't say that. At least of Windows. For some weird technical/historical reasons, the JVM ignores the fact that Unicode is the recommended encoding on Windows for 'all new applications' and instead always acts as if the legacy encoding configured as fallback for legacy apps is the 'platform default'. Nov 20, 2015 at 16:06
  • 7
    I would even go as far as saying that if your Java app does not explicitly specify encodings every time it's reading or writing to files/streams/resources, it's broken, because it can not ever work reliably then. Nov 20, 2015 at 16:08
14

For Java 7+ doc you can use this:

BufferedReader reader = Files.newBufferedReader(path, StandardCharsets.UTF_8);

Here are all Charsets doc

For example if your file is in CP1252, use this method

Charset.forName("windows-1252");

Here is other canonical names for Java encodings both for IO and NIO doc

If you do not know with exactly encoding you have got in a file, you may use some third-party libs like this tool from Google this which works fairly neat.

8

Since Java 11 you may use that:

public FileReader(String fileName, Charset charset) throws IOException;
1

FileInputStream with InputStreamReader is better than directly using FileReader, because the latter doesn't allow you to specify encoding charset.

Here is an example using BufferedReader, FileInputStream and InputStreamReader together, so that you could read lines from a file.

List<String> words = new ArrayList<>();
List<String> meanings = new ArrayList<>();
public void readAll( ) throws IOException{
    String fileName = "College_Grade4.txt";
    String charset = "UTF-8";
    BufferedReader reader = new BufferedReader(
        new InputStreamReader(
            new FileInputStream(fileName), charset)); 

    String line; 
    while ((line = reader.readLine()) != null) { 
        line = line.trim();
        if( line.length() == 0 ) continue;
        int idx = line.indexOf("\t");
        words.add( line.substring(0, idx ));
        meanings.add( line.substring(idx+1));
    } 
    reader.close();
}
0

For another as Latin languages for example Cyrillic you can use something like this:

FileReader fr = new FileReader("src/text.txt", StandardCharsets.UTF_8);

and be sure that your .txt file is saved with UTF-8 (but not as default ANSI) format. Cheers!

1
  • there's no other parameter than the file path! Aug 29, 2022 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.