7

I have been following an example of Bayesian classifiers according to the book of Lantz entitled "Machine Learning with R". The case is a spam classifier that works with the data of the following link:

http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/

In the code I have a problem in this part:

sms_train<-DocumentTermMatrix(sms_corpus_train,list(dictionary=sms_dict))
sms_test<-DocumentTermMatrix(sms_corpus_test,list(dictionary=sms_dict))

because it says that I should use the following instruction:

sms_dict <- Dictionary(findFreqTerms(sms_dtm_train, 5))

The problem is that the Dictionary() function has been deprecated from new versions of tm. What I should do to accomplish what the books says:

A dictionary is a data structure allowing us to specify which words should appear in a document term matrix. To limit our training and test matrixes to only the words in the preceding dictionary, use the following command

I have done the following:

sms_dict<-findFreqTerms(sms_dtm_train,5)
sms_train<-DocumentTermMatrix(sms_corpus_train,list(dictionary=sms_dict))
sms_test<-DocumentTermMatrix(sms_corpus_test,list(dictionary=sms_dict))

But I am sure that I am not limiting the test matrices at it says in the book. Even though the code is working, it does not give me the right results. What can I modify in this case?

The complete code for tracking purposes is the following:

sms_raw<-read.csv("sms_spam.csv",stringsAsFactors=FALSE)
install.packages("tm")
library(tm)
sms_corpus<-Corpus(VectorSource(sms_raw$text))
corpus_clean<-tm_map(sms_corpus,content_transformer(tolower))
corpus_clean<-tm_map(corpus_clean,removeNumbers)
corpus_clean<-tm_map(corpus_clean,removeWords,stopwords())
corpus_clean<-tm_map(corpus_clean,stripWhitespace)
sms_dtm<-DocumentTermMatrix(corpus_clean)
sms_raw_train<-sms_raw[1:4169,]
sms_raw_test<-sms_raw[4170:5559,]
sms_dtm_train<-sms_dtm[1:4169,]
sms_dtm_test<-sms_dtm[4170:5559,]
sms_corpus_train<-corpus_clean[1:4169]
sms_corpus_test<-corpus_clean[4170:5559]
sms_dict<-findFreqTerms(sms_dtm_train,5)
sms_train<-DocumentTermMatrix(sms_corpus_train,list(dictionary=sms_dict))
sms_test<-DocumentTermMatrix(sms_corpus_test,list(dictionary=sms_dict))
convert_counts<-function(x){
x<-ifelse(x>0,1,0)
x<-factor(x,levels=c(0,1),labels=c("No","Yes"))
return(x)
}
sms_train<-apply(sms_train,MARGIN=2,convert_counts)
sms_test<-apply(sms_test,MARGIN=2,convert_counts)
library(e1071)
sms_classifier<-naiveBayes(sms_train,sms_raw_train$type)
sms_test_pred<-predict(sms_classifier,sms_test)
install.packages("gmodels")
library(gmodels)
CrossTable(sms_test_pred,sms_raw_test$type,prop.chisq=FALSE,prop.t=FALSE,dnn=c('predicted','actual'))

Thanks

2
  • 2
    I think you have used findFreqTerms correctly as a dictionary. Your way fully replaces the former Dictionary() function, check this answer link. Could you please explain more clearly how your expected output differs from the one you currently get? This will make it easier to track down the issue. Nov 10, 2017 at 16:46
  • Aren't this [1:4169,] and [4170:5559] actually limiting your test matrices? Since you use a big portion for the training and later you have your smaller test set. But maybe I am completely off since I don't understand your code. It is hard to read with no proper spacing and unlogic variable names.
    – drmariod
    Nov 16, 2017 at 10:15

1 Answer 1

0

I had the same issue and solved it by doing:

CrossTable(sms_test_pred[["class"]], sms_raw_test$Type, prop.chisq = FALSE, prop.t = FALSE, dnn = c('predicted','actual'))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.