0

Similar to this SO member, I've been looking for a simple package in R that filters out words that are non-English. For example, I might have a list of words that looks like this:

Flexivel
eficaz
gut-wrenching
satisfatorio
apropiado
Benutzerfreundlich
interessante
genial
cool
marketing
clients
internet

My end goal is to simply filter out the non-English words from the corpus so that my list is simply:

gut-wrenching
cool
marketing
clients
internet

I've read in the data as a data.frame, although it will subsequently be transformed into a corpus and then a TermDocumentMatrix in order to create a wordcloud using wordcloud and tm.

I am currently using the package textcat to filter by language. The documentation is a bit above my head, but seems to indicate that you can run the command textcat on lists. For example, if the data above was in a data.frame called df with a single column called "words", I'd run the command:

library(textcat)
textcat(c(df$word))

However, this has the effect of reading the entire list of words as a single document, rather than looking at each row and determining it's language. Please help!

4
  • 1
    I voted to close because this question is asking for a package recommendation. Maybe tweak the question to how you would do this in R and show you did some leg work. Also eliminate the bonus question. Aug 26, 2013 at 16:08
  • Define "English words." (Hint: you can't.) Aug 26, 2013 at 16:47
  • @TylerRinker - Thanks for the input. I found the cldr package that uses Chrome to detect languages, but that seems to apply a probabilistic judgment to generate a guess about the "top three possible languages". This is a bit more sophisticated than what I need, so I was looking for a simpler dictionary based approach. I'll keep scouting, and make the question more specific once I find an option. (PS, I love qdap :))
    – roody
    Aug 26, 2013 at 16:57
  • @roody the problem is textcat (as Carl alludes to) is going to tell you origins so only gut wrenching is an English word. One approach may be to use a dictionary of English words (maybe like qdap's DICTIONARY[, 1] data set). Then stem this and the list of words and use %in% or a lookup environment to determine the words that are "English" Aug 26, 2013 at 17:52

1 Answer 1

0

For a dictionary search you could use aspell:

txt <- c("Flexivel", "eficaz", "gut-wrenching", "satisfatorio", "apropiado",
  "Benutzerfreundlich", "interessante", "genial", "cool", "marketing",
  "clients", "internet")

fn <- tempfile()
writeLines(txt, fn)
result <- aspell(fn)

results$Original gives the non-matching words. From those you can select the matching words:

> result$Original
[1] "Flexivel"           "eficaz"             "satisfatorio"      
[4] "apropiado"          "interessante"       "Benutzerfreundlich"
> english <- txt[!(txt %in% result$Original)]
> english
[1] "gut-wrenching" "genial"        "cool"          "marketing"    
[5] "clients"       "internet"

However, as Carl Witthoft indicates you can not be sure that these are actually English words. 'cool', 'marketing' and 'internet' are also valid Dutch words for example.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.