4

So I need to find nearest neighbors of a given row in pyspark DF using euclidean distance or anything. the data that I have 20+ columns, more than thousand rows and all the values are numbers.

I am trying to oversample some data in pyspark, as mllib doesn't have inbuilt support for it, i decided to create it myself using smote.

my approach till now has been to convert all the categorical distance into index using stringtoindex so that i can find the euclidean distance and neighbors and hence perform smote.

I am fairly new to spark and ml. Any help would be appreciated.

2

1 Answer 1

2

Not tried but Ive found this script: https://github.com/jakac/spark-python-knn/blob/master/python/gaussalgo/knn/knn.py

If your data are dataframe, you should first merge your column into a vector with vectorASsembler https://spark.apache.org/docs/latest/ml-features.html#vectorassembler, then use df.select("id", "yourColumnVector")

The library I provided seems to work only with rdd, so you should convert your dataframe to RDD using df.rdd

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.