1

I'm devising a system for determining the human language of a given text. It works by storing a dictionary for each language of interest and subsequently assessing whether the user input corresponds to any words stored by the dictionary. The language whose dictionary has the most hits is the winner.

I guess that in order for these two entries, the user's word and the word that comes from the "word list" file, to match the encoding will have to be the same, that's what I'm trying to fix.

I took the word lists from this site.

When I use the ones from "ASCII" in my code everything works, but when I use the ones from "Unicode" nothing works.

This is disconcerting to me because I don't want the programme to get some input that's encoded in the wrong way (in a way that conflicts with my word-list data structures) and then fail.

For this reason I want to standardize all input with a particular encoding. I was thinking that "Unicode" would be better because, as this is a system for determining the natural language of a text I might get some Greek or Russian or Chinese characters, and from my understanding "ASCII" is highly non-standardized.

I'm currently using the console of Eclipse to write the input.

This is how I read in files:

//BufferedReader br = new BufferedReader( new InputStreamReader(new FileInputStream( dir.toString() ), "UTF-8") );

BufferedReader br = new BufferedReader( new FileReader( dir.toString() ) );
String line = null;

BloomFilter<String> bloomFilter;
if (word_holding_directory_path.toLowerCase().contains("/de/")) 
{
    bloomFilter = de_bloomFilter;
} 

This is how I read in user input:

//Scanner in = new Scanner( System.in , "UTF-8");
Scanner in = new Scanner(System.in);

System.out.println("Please enter a sentence: ");

String[] input_text = in.nextLine().split("\\s");

As you can see, I tried to force the encoding to be UTF-8, (that's the same thing as Unicode, isn't it?), but as it wasn't working I commented it out.

This is how I compare the words:

for (String word : input_text) 
{
    String normalized = word.trim().toLowerCase();
    if (words.contains(normalized)) 
    {
        ++count;
    }
}

The full programme is here on github it's pretty short and fairly explicitly commented.

6
  • 1
    One approach I have seen in n-gram language identification is to have one model each for every language/encoding pair. So one model for Russian-UTF8, another for Russian-UTF16LE, another for Russian-KOI8R. Granted, if you have very large models, keeping just one in Unicode and deriving the others on the fly is probably better.
    – tripleee
    May 19, 2015 at 7:19
  • 1
    Unicode and UTF-8 are not the same thing. Unicode is an abstract encoding which can be realized as UTF-8, UTF-16, UTF-32, and a variety of other encodings; though commonly, UTF-8 is the default these days.
    – tripleee
    May 19, 2015 at 7:22
  • 1
    What you should fix is the way you read the file, not the way you read from System.in.
    – JB Nizet
    May 19, 2015 at 7:22
  • 1
    Based on a quick glance at se.zip, the dictionaries are in UTF-16LE.
    – tripleee
    May 19, 2015 at 7:24
  • @tripleee so I should enforce encoding as UTF-16LE? May 19, 2015 at 7:28

1 Answer 1

1

The dictionaries you link to seem to be in UTF-16LE, not UTF-8. You should fix the encoding parameter in the InputStreamReader instantiation accordingly.

Unicode and UTF-8 are most emphatically not the same thing; and indeed, saying that text is "Unicode" without mentioning the encoding is not precise enough.1

(You should be able to guess that a ZIP file which is several years old might use UTF-16LE, which is still the default on Windows for many things. New resources would typically be UTF-8, even on Windows.)

Reading one string from an UTF-16 file and another containing the same text from the console with the correct console encoding will produce two Java strings which are equal. On the other hand, if the encoding on one of the input streams is incorrect, what you end up with in the string will be essentially random bogus. (In the "train wreck" scenario, you have different encoding errors on both inputs, and just coincidentally get two equal strings when in fact the two strings are supposed to contain different text.)

(Not sure if UTF-8 is generally correct for the console, or perhaps specifically in your environment -- I'm not a Java programmer.)


1 Briefly, the abstract Unicode string

U+0066 U+00F6 U+0072

(which represents the Swedish word för) would be represented as

0x66 0xC3 0xB7 0x72

in UTF-8 (notice how the plain ASCII characters map to identity representation), and

0x66 0x00 0xF6 0x00 0x72 0x00

in UTF-16LE (where each pair of bytes is one 16-bit sequence with the MSB set to zero, while the LSB accommodates the entire significant part of the representation).

In plain ASCII, this string cannot be represented; way back in time, it would have been written as

0x66 0x7C 0x72

where 0x7C is properly the pipe character | which was locally mapped to the glyph ö in hardware. (Correspondingly, if you were using resources in English which were supposed to contain a proper pipe character, that too would have been rendered as ö; so a Unix pipeline line grep cat food | xxd would display as grep cat food ö xxd.)

Somewhat later in time, ISO-8859-1 became popular, and this string would be represented as

0x66 0xFC 0x72

That's certainly simple and efficient. Why isn't this how Unicode represents it, too? Because there are only 256 characters in an 8-bit encoding, and Unicode is much larger. You cannot represent 上海市 or машина at all.

14
  • but what about the user input? It's not trying to predict the input of files, my goal is to disregard that and enforce them all to conform to the same standard so I can compare them, you know what I mean? Is it possible to do that? May 19, 2015 at 7:32
  • 1
    Reading it into memory will use whatever Unicode representation Java prefers (I believe that would be UTF-16 internally, but it doesn't really matter) and thus as long as you have compatible representations, your comparisons will work. So just to reiterate, just make sure you have the correct encoding for the inputs you read. If your console is UTF-8 then you should use that for reading the user input.
    – tripleee
    May 19, 2015 at 7:35
  • ahh. so as long ad the encodings are comparable it will work. but it seems that whatever is the preference for console input in Eclipse and the Unicode I got from that site, they are not compatible. May 19, 2015 at 7:36
  • 1
    Reading UTF-16 into memory as UTF-8 should simply break with an error message (where I come from, which admittedly is not Java). Using the correct encoding when reading the inputs into memory will make them comparable -- in memory, they are just Java strings.
    – tripleee
    May 19, 2015 at 7:38
  • so then it doesn't really matter if I store them as ASCII or Unicode? May 19, 2015 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.