I am looping over text to find and count specific words from various dictionaries. I use two FOR loops which are extremely slow and takes days to complete. Reproducible code below:
library(stringr)
#Sample data
tweets=data.frame(id=c(1,2,3),text=c("This is a tweet that contains word1",
"And here you can find word1 and word2 word2",
"And here is only one word3 and one word3a"))
words=data.frame(id=c(1,2,3),word=c("word1","word2","word3"))
for(i in 1:nrow(tweets)){
for(j in 1:nrow(words)){
term = paste("\\<",words[j,2],"\\>", sep="")
if (str_count(tweets[i,2], term) != 0) {
tmp <- data.frame(id=tweets[i,1],termfound=words[j,2],count=str_count(tweets[i,2], term), row.names=NULL)
message("ID ",tweets[i,1]," - Word '",words[j,2],"' found ",str_count(tweets[i,2], term)," times")
#sqlSave(myconn, tmp, "DataTable", append=T, rownames=F)
}
}
}
NOTES:
I have ~1M lines of text and ~25,000 words I am counting.
The Message line is just for debugging.
The final values are written to SQL - line commented out as it is not reproducible.
Any way to improve on this? I was thinking an APPLY function???
Cheers B
tm
package useful, it has many functions for this kind of text mining. The vignette demonstrates how to use most of them. It uses document term matrices as sparse matrices which are quite fast to work with.stringi
in your lifestringi
is written in C and virtually everything is vectorized, making it great for nesting and passing to other package functions.