I am trying to build a recommendation engine based on collaborative filtering using apache Spark. I have been able to run the recommendation_example.py
on my data, with quite good result. (MSE
~ 0.9). Some of the specific questions that I have are:
- How to make recommendation for the users who have not done any activity on the site. Isn't there some API call for popular items, which would give me the most popular items based on user actions. One way to do is to identify the popular items by ourselves, and catch the
java.util.NoSuchElementException
exception, and return those popular items. - How to reload the model, after some data has been added in the input file. I am trying to reload the model using another function, which tries to save the model, but it gives error as
org.apache.hadoop.mapred.FileAlreadyExistsException
. One way to do is to listen for the incoming data on a parallel thread, save it usingmodel.save(sc, "target/tmp/<some target>")
and then reload the model after significant data has been received. I am lost here, how to achieve that.
It would be very helpful, if I could get some direction here.