I am attempting to filter out the stop words out of an RDD of words from a .txt
file.
// Creating the RDDs
val input = sc.textFile("../book.txt")
val stopWordsInput = sc.textFile("../stopwords.csv")
val stopWords = stopWordsInput.map(x => x.split(","))
// Create a tuple of test words
val testWords = ("you", "to")
// Split using a regular expression that extracts words
val wordsWithStopWords = input.flatMap(x => x.split("\\W+"))
The code above all makes sense to me and seems to work fine. This is where I'm having trouble.
//Remove the stop words from the list
val words = wordsWithStopWords.filter(x => x != testWords)
This will run but doesn't actually filter out the words contained in the tuple testWords
. I'm not sure how to test the words in wordsWithStopWords
against every word in my tuple testWords