0

I am trying to have a program that normalizes my text, it removes multiple empty spaces, it prints the other characters from the original file, and also put spaces and start and ending symbols.

So the conversion, after I write the txt file and open it, I see this content:

numa situaã § ã £ o de emergãªncia mã © dica

as you can see there are some weird characters that I don't want, maybe it's because of Encoding?? This is a text in my language, Portuguese.

Here is my code, how can I fix it?

public static void main(String[] args) throws IOException {
        
        Charset encoding = Charset.defaultCharset();
        
        InputStream in = new FileInputStream(new File("data.txt"));
        Reader reader = new InputStreamReader(in, encoding);
        Reader buffer = new BufferedReader(reader);
        StringBuilder normalizedLanguage = new StringBuilder("<");
        int r;
        while ((r = buffer.read()) != -1) {
            char ch = (char) r;
         
            


            boolean newline = false;
            boolean hasLetterBefore = false;
            boolean hasLetterAfter = false;
            char symbol = '-';
            int lines = 0;

            if (newline)
            {
                normalizedLanguage.append("\n<");
            }


            if (ch == '\r' || ch == '\n' )
            {
                lines++;
                normalizedLanguage.append(">");
                newline = true;
                hasLetterBefore = false;
                

            }
            else if (Character.isLetterOrDigit(ch))
            {
                if (hasLetterBefore == true)
                {
                    normalizedLanguage.append(Character.toString(symbol) + Character.toString(Character.toLowerCase(ch)));
                }else{
                    normalizedLanguage.append(Character.toString(Character.toLowerCase(ch)));
                }

               
                newline = false;
                hasLetterBefore = true;
            }
            else if (ch == ' ')
            {
                normalizedLanguage.append(Character.toString(ch));
                newline = false;
                hasLetterBefore = false;
            }
            else if (ch == '\t')
            {
                System.out.println("Tab detected: " + ch);
                newline = false;
                hasLetterBefore = false;
            }
            else
            {
                //Símbolos, entre outros..
                if (!hasLetterBefore)
                {
                    normalizedLanguage.append(" " + Character.toString(ch) + " ");
                }
                else
                {
                    symbol = ch;
                }
                newline = false;
                
            }


        }

        String normalizedLanguageString = normalizedLanguage.toString().trim().replaceAll(" +", " ");
        
        PrintWriter out = new PrintWriter("data_after.txt");
        
        out.println(normalizedLanguageString);
        out.close();
        
        buffer.close();
        reader.close();
        in.close();

    }

Thank you very much in advance ;)

1 Answer 1

0

The problem got solved using another Charset Encoding :)

Change this line:

Charset encoding = Charset.defaultCharset();

To:

Charset encoding = Charset.forName("UTF8");

Thank you very much anyways

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.