I'm devising a system for determining the human language of a given text. It works by storing a dictionary for each language of interest and subsequently assessing whether the user input corresponds to any words stored by the dictionary. The language whose dictionary has the most hits is the winner.
I guess that in order for these two entries, the user's word and the word that comes from the "word list" file, to match the encoding will have to be the same, that's what I'm trying to fix.
I took the word lists from this site.
When I use the ones from "ASCII" in my code everything works, but when I use the ones from "Unicode" nothing works.
This is disconcerting to me because I don't want the programme to get some input that's encoded in the wrong way (in a way that conflicts with my word-list data structures) and then fail.
For this reason I want to standardize all input with a particular encoding. I was thinking that "Unicode" would be better because, as this is a system for determining the natural language of a text I might get some Greek or Russian or Chinese characters, and from my understanding "ASCII" is highly non-standardized.
I'm currently using the console of Eclipse to write the input.
This is how I read in files:
//BufferedReader br = new BufferedReader( new InputStreamReader(new FileInputStream( dir.toString() ), "UTF-8") );
BufferedReader br = new BufferedReader( new FileReader( dir.toString() ) );
String line = null;
BloomFilter<String> bloomFilter;
if (word_holding_directory_path.toLowerCase().contains("/de/"))
{
bloomFilter = de_bloomFilter;
}
This is how I read in user input:
//Scanner in = new Scanner( System.in , "UTF-8");
Scanner in = new Scanner(System.in);
System.out.println("Please enter a sentence: ");
String[] input_text = in.nextLine().split("\\s");
As you can see, I tried to force the encoding to be UTF-8, (that's the same thing as Unicode, isn't it?), but as it wasn't working I commented it out.
This is how I compare the words:
for (String word : input_text)
{
String normalized = word.trim().toLowerCase();
if (words.contains(normalized))
{
++count;
}
}
The full programme is here on github it's pretty short and fairly explicitly commented.
se.zip
, the dictionaries are in UTF-16LE.UTF-16LE
?