0

The following is the input file to Apache spark processing. each record contains four fields as shown.

User ID     Movie ID         Rating      Date
6             9217            2       2005-04-28 00:00:00.000
6             9254            4       2005-04-20 00:00:00.000
6             9330            4       2004-11-17 00:00:00.000
6             9330            5       2004-09-27 00:00:00.000
6             1615            4       2004-09-15 00:00:00.000
6             1659            3       2005-02-18 00:00:00.000
6             9254            4       2005-10-26 00:00:00.000
6             9217            3       2005-11-25 00:00:00.000
6             9217            3       2004-09-15 00:00:00.000

from each of the records, i need to extract Movie ID and Rating into key/value pairs as ex: (Movie ID,Rating ). i went though the documentation of Apache spark and done some surfing. But couldn't found the relevant stuff. Any suggestions would be appreciated.

1
  • Think about how you would parse this file without Spark, and then see how you can apply that to an RDD.
    – Mike Park
    Mar 9, 2015 at 15:26

2 Answers 2

2

You can do something like this:

val text= sc.textFile("movies.txt")
val LogEntry = """^\s*([0-9]+)\s+([0-9]+)\s+(\d{1})\s+(.*)""".r
val logEntries = text.flatMap(line => line match {
  case LogEntry(userid,movieid,rating,date) => Some((userid.toInt,movieid.toInt,rating.toInt,date))
  case _ => None
}).cache()
val movieTotalRating = logEntries.map(line => line match { case (userid,movieid,rating,date) => (movieid,rating)}).reduceByKey((rating1, rating2) => rating1 + rating2)

Notice, I use the .flatMap on the Optional return type to remove rows which are not matching the RegEx (like the header row)

By caching the logEntries you can compute several statistics (like movieTotalRating here) from the same parsed dataset.

0

I could do it on sample row using pattern matching. But not sure about how to apply to all the lines of input file to generate desired key/value pair (movie Id, Rating).

scala> val LogEntry = """^\s*([0-9]+)\s+([0-9]+)\s+(\d{1})\s+(.*)""".r
LogEntry: scala.util.matching.Regex = ^\s*([0-9]+)\s+([0-9]+)\s+(\d{1})\s+(.*)

scala> val LogEntry(userid,movieid,rating,date) = "6             9217            2       2005-04-28 00:00:00.000"
userid: String = 6
movieid: String = 9217
rating: String = 2
date: String = 2005-04-28 00:00:00.000

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.