1

I am new to Spark ML. I want to recommend movies to users using Apache Spark ML. I learned here that we can recommend movies based on rating by a user.

My question is can we include other features for recommendation, say, his age, country, genre of movie, likes, etc. For example, we have users 'U1', 'U2', 'U3' and 'U4' all of them watching Movie 'M'. U1 of age 22 living in America also watched movie M1, M2 and M3 and U2 of age 50 living in Australia watched movie M1, M4 and M5. Now, I would like to recommend U3 of age 24 living in America movies M1, M2 and M3. Also Recommend movies M1,M4 and M5 to U4 of age 21 living in Australia.

Basically, I want to provide some weights to age and country. How can we achieve this with Spark ml(say using ALS)?

2 Answers 2

5

This problem of introducing a context information can be addressed in a few different ways:

  • if you have to stick to Spark and 2D (user x item) recommender: then go for a contextual post/pre filtering. As you have the MF with ALS solver already there, you can use it as model base recommender in two ways for what you are trying to achieve: (1/pre-filtering) group your data by you context (e.g. age ranges, country) before creating 2D MF-ALS recommender for each of group or (2/post-filtering) create on single 2D user-item recommender an then do the post-filtering based on your contextual information after getting recommendations from your model. In the second option, you may try not only the boolean filtering but also boosting score value based on the contextual match, e.g. boost based on the avg(age of the watchers).
  • use contextual information directly into a model based recommender. This one cannot be directly applied with core/MLlib Spark matrix factorization method. This one is prepared for 2D, user-item actions. But to overcome this you can use different model-based method. I would advice going for Factorization Machines as it is a successful model for context-aware recommendations and it's very intuitive, especially in data preparation. You can see introduction from my slides here: Intro to FM slides. There is a package for Apache Spark - Spark libFM Package, but cannot say anything about it - is it really working or still maintained (?). But for sure can advice main libfm.org, python - fastFM and very similar+fast implementation lightfm from Lyst, which can be useful for your problem.
  • do kNN recommender where you introduce your own similarity measures while comparing user2user. It's heuristic base, where you can boost similarity score where the contextual information about both user match - e.g. using sth like: abs(u1.age - u2.age)/(max(age) - min(age), country, city or region distance between u1 and u2.

I would also advice to read the chapter from Recommender Systems Handbook[6] about introducing contextual information to recommender systems. And of course the whole book!

[6]: Kantor, Paul B. Recommender systems handbook. Eds. Francesco Ricci, Lior Rokach, and Bracha Shapira. Berlin, Germany:: Springer, 2015.

0

I made a project on movie recommendation system using Apache Spark MLlib. I had help from codementor. You can find the tutorial here.

2
  • Thanks. Its a nice blog, but it still doesn't solve my query about how to use multiple features(like age, sex, etc) apart from rating for recommending content.
    – Rohan
    Jun 27, 2017 at 11:02
  • 1
    @Rohan were you able to figure out a neat way of implementation for this? Dec 4, 2020 at 5:12

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.