2

I am attempting to filter out the stop words out of an RDD of words from a .txt file.

// Creating the RDDs
val input = sc.textFile("../book.txt")
val stopWordsInput = sc.textFile("../stopwords.csv")
val stopWords = stopWordsInput.map(x => x.split(","))

// Create a tuple of test words
val testWords = ("you", "to")

// Split using a regular expression that extracts words
val wordsWithStopWords = input.flatMap(x => x.split("\\W+"))

The code above all makes sense to me and seems to work fine. This is where I'm having trouble.

//Remove the stop words from the list
val words = wordsWithStopWords.filter(x => x != testWords)

This will run but doesn't actually filter out the words contained in the tuple testWords. I'm not sure how to test the words in wordsWithStopWords against every word in my tuple testWords

1
  • I don't see the usage of testWords.
    – eliasah
    Jan 12, 2017 at 16:59

3 Answers 3

5

You can use a Broadcast variable to filter with your stopwords RDD :

// Creating the RDDs
val input = sc.textFile("../book.txt")
val stopWordsInput = sc.textFile("../stopwords.csv")

// Flatten, collect, and broadcast.
val stopWords = stopWordsInput.flatMap(x => x.split(",")).map(_.trim)
val broadcastStopWords = sc.broadcast(stopWords.collect.toSet)

// Split using a regular expression that extracts words
val wordsWithStopWords: RDD[String] = input.flatMap(x => x.split("\\W+"))
wordsWithStopWords.filter(!broadcastStopWords.value.contains(_))

Broadcast variables allow you to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. They can be used, for example, to give every node a copy of a large input dataset in an efficient manner (in this case as well).

4

You are testing a String over a a tuple ("you", "to"), which will always be false.

Here is what you want to try:

val testWords = Set("you", "to")
wordsWithStopWords.filter(!testWords.contains(_))

// Simulating the RDD with a List (works the same with RDD)
List("hello", "to", "yes") filter (!testWords.contains(_))
// res30: List[String] = List(hello, yes)
9
  • That doesn't seem to work. Theres an error with both ! and .contains. Jan 12, 2017 at 16:49
  • 1
    I like this! :-) You can also write it like this: wordsWithStopWords filterNot testWords. Jan 12, 2017 at 16:50
  • 2
    @JakeHenningsgaard The answer has been changed slightly (notice the Set method being called on the first line of the snippet), perhaps you can try it again. In case you are coming from Python, please note that Scala has a very different concept of tuples. Jan 12, 2017 at 16:53
  • 1
    @stefanobaghino spark doesn't implement filterNot.
    – eliasah
    Jan 12, 2017 at 16:58
  • 1
    Let's say that it's safer to use a broadcast variable instead of collect even if this is a plausible solution.
    – eliasah
    Jan 12, 2017 at 17:05
0

Using subtractByKey:

// Creating the RDDs
val input = sc.textFile("../book.txt")
val stopWordsInput = sc.textFile("../stopwords.csv")

// Split using a regular expression that extracts words from input RDD
val wordsWithInput = input.flatMap(x => x.split("\\W+"))


//Converting above RDDs to lowercase
val lowercaseInput = wordsWithInput.map(x => x.toLowerCase())
val lowercaseStopWordsInput = stopWordsInput.map(x => x.toLowerCase())

//Creating a tuple(word, 1) using map for above RDDs
val tupleInput = lowercaseInput.map(x => (x,1))
val tupleStopWordsInput = lowercaseStopWordsInput.map(x => (x,1))

//using subtractByKey
val tupleWords = tupleInput.subtractByKey(tupleStopWordsInput)

//to have only words in RDD
val words = tupleWords.keys

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.