1

I have application that uses Spark to do some data processing and saves it as RDD; I'm initializing SparkContext with "local" attribute. I want to use Mahout on this dataset, but what is the best way to share this data between 2 applications?

I'm new to big data ecosystem, so answer is not really obvious to me.

p.s. I'm also installed Hadoop locally with help of this guide http://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/ and able to run it.

1 Answer 1

1

Since your data is in an RDD it can only be accessed from the context that created it unless you save it to a file or some other persistence mechanism.

Mahout is a library like MLlib so as long as you make the Spark context available to the code it should work fine.

Mahout has higher level data type that support general linear algebra so you may need to convert the raw rdd into something like Mahout's RDD backed DrmLike[Int], which is the primary "DistributedRowMatrix" type. This will depend on what your RDD contains and what you want to do with it so you'll have to explain a little more.

Only a couple lines of code are typically required to take a DRM and convert it into MLlib RDD of Vectors for processing by KMeans. The opposite direction is also pretty easy.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.