Questions tagged [apache-spark-mllib]
MLlib is a machine learning library for Apache Spark
2,229
questions
0
votes
0
answers
8
views
PySpark MLLib APproximate nearest neighbour search for multiple keys
I want to use ANN from PySpark. I have a DataFrame of 100K keys for which I want to perform top-10 ANN searches on an already transformed Spark DataFrame. But it seems that API of ...
0
votes
1
answer
24
views
How to implement Imputation in spark
I want to perform Mean, Median, Mode and use user defined value for imputation on spark dataframe
Is there any best way to do these in java.
For Example, suppose I am having these five columns and ...
0
votes
1
answer
24
views
Can we create custom Estimators
I want to create my own Estimator for Spark ml pipeline purpose so that I can use my own custom business logic.
If any one can guide me in this using Java will be very helpful.
Update:
I created one ...
0
votes
0
answers
17
views
Tuning `CrossValidator` spark job performance
I am running a 3-fold cross validation of an ML pipeline that utilizes GBTClassifier as the final step. It takes 18 hours to run and I am looking for feedback into how to improve the performance as I ...
0
votes
0
answers
13
views
Is not all pyspark API work with distributed?
I used under code VectorAssembler and StandardScaler to standardization.
But when VectorAssembler working, Spark job was not shown and very slow.
I can't know how many tasks succeeded, duration and so ...
0
votes
1
answer
29
views
Best way to Create a custom Transformer In Java spark ml
I am learning Big data using Apache spark and I want to create a custom transformer for Spark ml so that I can execute some aggregate functions or can perform other possible operation on it
0
votes
0
answers
11
views
Pyspark: 'HashingTF' object has no attribute 'setInputCol'
I'm using pyspark 3.2.1 and I'm trying to create a HashingTF object and set input and output columns.
Neither
hashingTF = HashingTF(numFeatures=20).setOutputCol('output'),
nor
hashingTF = HashingTF(...
0
votes
0
answers
17
views
I am new with Pyspark, I am trying onehotencoding in iot dataset for deeplearning implementation
When I am trying to fit the pipeline, I am getting a error like this
Py4JJavaError: An error occurred while calling o1380.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task ...
0
votes
1
answer
28
views
Spark Mllib DecisionTree skewed task runtime
I'm using apache spark mllib to learn a regression tree for a quite large dataset and I have found what I think is an abnormal behviour in one of the algorithm stages.
As you can see in the next ...
0
votes
0
answers
10
views
For text classification in Pyspark, pass train vocabulary to test or use test vocabulary as features?
I need to implement logistic regression to train and test dataset in Pyspark, both of them are text file. For train dataset, the top 5 vocabularies would be its features, so I implemented Tokenizer, ...
0
votes
1
answer
19
views
Apply vectors.Dense() to an array float column in pyspark 3.2.1
In order to apply PCA from pyspark.ml.feature, I need to convert a org.apache.spark.sql.types.ArrayType:array<float> to org.apache.spark.ml.linalg.VectorUDT
Say I have the following dataframe :
...
-1
votes
0
answers
48
views
DECISION TREE CLASSIFIER MLLIB PYSPARK - COULD NOT ACCESS LEAFCOL
I am actually trying to fit a DecisionTreeClassifier using MLLIB and trying to access LeafCols = 'leafId' of each sample but unfortunately it is throwing an error stating there is no argument LeafCols ...
0
votes
1
answer
27
views
Implementing RL algorithm on apache spark
I want to run RL algorithm on Apache Spark. However, RL does not exists in Spark's MLib.
Is it possible to implement it? any links may help.
Thank you in advance
0
votes
0
answers
23
views
Spark Java: How can we access the p Values in UnivariateFeatureSelector
I am using Spark 3.1.3 and I am trying to take the pValues from the result dataframe of UnivariateFeatureSelector.
UnivariateFeatureSelector selector = new UnivariateFeatureSelector()
....
0
votes
0
answers
12
views
Measure ML model prediction speed in Spark
In this Jupyter notebook I used 70% of the dataset as training and 30% as testing. I perform fit for the training part and transform for the testing part.
dt = DecisionTreeClassifier(featuresCol = '...
0
votes
1
answer
19
views
matrix factorization model returning much smaller dataframe after predicting ratings in pyspark
I'm trying to create a product recommender with the code below. I'm using matrix factorization from spark ml. I have data that has a customer_id, product_id, and a numeric rating value that has been ...
0
votes
0
answers
13
views
user_id mismatch in Pyspark ALS
I trained the ALS model on a set of 450000 unique user_id's. Following which I extracted the user_matrix from it using model.userMatrix, then I did inner join this dataframe with my train dataframe on ...
0
votes
0
answers
20
views
Why is PySpark LinearSVC only predicts 1
This is the dataframe df and the ratio of rows with label equal to 0 is 0.13217391304347825
+---------------------------------------+-----+
|features |label|
+-----------...
0
votes
1
answer
56
views
How do I extract feature_importances from my model in SparklyR?
I would like to extract feature_importances from my model in SparklyR. So far I have the following reproducible code that is working:
library(sparklyr)
library(dplyr)
sc <- spark_connect(method = &...
0
votes
0
answers
19
views
How do you calculate p/t for thresholds argument in sparklyR ml_random_forest_classifier?
How do you calculate p/t for thresholds argument in sparklyR's ml_random_forest_classifier()? I have a binary classification model with original class probabilities of 0.9 and 0.1.
The documentation ...
0
votes
0
answers
25
views
pyspark LinearSVC getting 3 classes of labelcol where there is only 2 classe in my variable
I used to run my models ( LinearSVC+ randomforest + ... ) on a training dataset and it goes well.
last time i tried to re-train my model all my models work except for the LinearSVC.
its getting the ...
0
votes
1
answer
33
views
Generate sparse vector for all the column values in spark dataframe
column1
column2
1
1
1
0
1
0
0
0
Now I want to calculate the hash or sparse vector of all the values in column1 and column2
0
votes
1
answer
32
views
How to groupBy and perform data scaling over each and every group using MlLib Pyspark?
I have a dataset just like in the example below and I am trying to group all rows from a given symbol and perform standard scaling of each group so that at the end all my data is scaled by groups. How ...
0
votes
0
answers
22
views
SparkNLP is there a way to split embeddings pipeline from classifier in order to reuse the embeddings layer for multiple classifiers
I'm trying to create multiple classifiers for different tasks that use the same sentence embedding stage.
by reusing the same sentence embedding layer memory consumption would be reduced significantly ...
0
votes
0
answers
17
views
How to save out mllib GBTRegression in a parquet and load it back?
Summarize the problem:
I have trained a GBT Regression model using mllib package to serve it in production. The way we design the system right now requires all data needed to be saved in a MODEL_INPUT ...
0
votes
1
answer
54
views
Pyspark Pipeline Performance
Is there any performance difference between using 2 separate pipelines vs 1 combined pipeline?
For example, 2 separate pipelines:
from pyspark.ml import Pipeline
from pyspark.ml.feature import ...
0
votes
0
answers
12
views
How to convert a mllib.linalg.Matrix to a trait matrix (mllib.linalg.Matrices) in scala spark?
I have a org.apache.spark.mllib.linalg.Matrix, which has access to the following (limited) methods (https://spark.apache.org/docs/latest/api/scala/org/apache/spark/mllib/linalg/Matrix.html). I would ...
0
votes
0
answers
15
views
How to divide a org.apache.spark.mllib.linalg.Matrix by a constant in Spark Scala?
I have a org.apache.spark.mllib.linalg.Matrix that can be generated from the following code:
val dfDouble = Seq(
(1.0, 1.0, 1.0, 3.0),
(1.0, 2.0, 0.0, 0.0),
(1.0, 3.0, 1.0, 1.0),
(1.0, NaN, 0....
0
votes
0
answers
108
views
PySpark python issue: Py4JJavaError: An error occurred while calling o44.trainALSModel
I'm new to spark and was playing around with some code that utilized the pyspark mllib library but was unable to run it.
Here's the code:
import sys
from pyspark import SparkConf, SparkContext
from ...
0
votes
0
answers
107
views
K-prototype clustering in Pyspark
I am working on dataset which has 20 Million record.
Data contains more than 30 features of both data types(continuous and categorical).
Choose optimal K using Elbow method
cost = []
for cluster in ...
0
votes
0
answers
21
views
PySPark mllib BinaryClassificationEvaluator giving weird results
I trained a NaiveBayesModel with labels 1.0 and 0.0 and predicted on the test data. If I look manually the predictions look rather good.
predictions
But using the BinaryClassificationEvaluator gives ...
1
vote
0
answers
29
views
PySpark MLLib equivalent of sklearn DictVectorizer
I am converting a sklearn build to a spark build. My data looks like this:
+-------+--------------------+
| source| feature|
+-------+--------------------+
|7593648|Map(8495310 -> 9,...|...
0
votes
1
answer
55
views
How to convert a DataFrame to an Array of dense vectors?
How would I convert the following DataFrame
val df = Seq(
(5.0, 1.0, 1.0, 3.0, 7.0),
(2.0, 0.0, 3.0, 4.0, 5.0),
(4.0, 0.0, 0.0, 6.0, 7.0)).toDF("m1", "m2", "m3", &...
1
vote
0
answers
22
views
Spark Streaming analytics on a twiter dataset streamed on tcp port
I am streaming a dataset stored locally on a PC in batch sizes of 10 and this is done to TCP port 6100
on the other hand I am fetching these files from this port as JSON files with the format shown ...
0
votes
0
answers
22
views
Training 100000+ models parallely using MLLib in PySpark
I am developing a forecast module for a portfolio of 4000+ products (the count of products will increase as we expand the portfolio). We are currently experimenting with random forest & XGBoost ...
0
votes
1
answer
120
views
Adding custom metadata to DataFrame schema using iceberg table format
I'm adding custom metadata into the DataFrames schema in my PySpark application using StructField's metadata field
It worked fine when I wrote parquet files directly into s3.
The custom metadata was ...
0
votes
1
answer
42
views
Training/Test data with SparkML in Scala
I've been facing with an issue for the past couple of hours.
In theory, when we split data for training and testing, we should standardize the data for training independently, so as not to introduce ...
0
votes
0
answers
45
views
Multiple Evaluators in CrossValidator - Spark ML
Is it possible to have more than 1 evaluator in a CrossValidator to get R2 and RMSE at the same time?
Instead of having two different CrossValidator:
val lr_evaluator_rmse = new ...
2
votes
1
answer
234
views
Error while I am using DataFrame show method in Pyspark
I try to show the Pyspark Dataframe, and I encounter such error:
Py4JJavaError: An error occurred while calling o607.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: ...
0
votes
1
answer
117
views
ML Tuning - Cross Validation in Spark
I am looking the cross validation code example found in https://spark.apache.org/docs/latest/ml-tuning.html#cross-validation
It says:
CrossValidator begins by splitting the dataset into a set of ...
0
votes
0
answers
34
views
What is expected Input of _LightGBMClassifier().fit()?
Here's my Input
data
DataFrame[features: vector, label: double]
Here's my code
from mmlspark.lightgbm._LightGBMClassifier import _LightGBMClassifier
_LightGBMClassifier().fit(data)
Here's my output
---...
1
vote
1
answer
83
views
Importing a trained pipeline model from pyspark to scala?
Is it possible to load a trained pipeline model from a pyspark environment to scala ?
I am trying to do it but I am having this error
requirement failed: Error loading metadata: Expected class name ...
0
votes
0
answers
26
views
spark gbm classifier giving slightly different probabilities for same feature values in dataset
I'm trying to predict whether pairs of records are a match using a gbm classifier. So I do a self join in my data to create pairs of records side by side, then use the classifier to predict if they'...
0
votes
0
answers
106
views
Setting custom parameters for a Spark MLlib pipeline
By default a Spark MLlib pipeline has one parameter: stages. The Params mechanism is brought in to the Pipeline class from the Params trait (via the abstract PipelineStage). In theory, it should be ...
1
vote
0
answers
57
views
avoid pivot when preprocessing data for vector assembler
I have the pyspark dataframe below called id_predictions_df. I'm pivoting it to get it in the pivot_df2 form show below, so I can then run vectorassembler on it and get it in a form that I can ...
4
votes
1
answer
80
views
What is the Exact Apache-Spark NA Treatment Difference Pandas vs MLLib for Covariance Computation?
I recently observed significant differences in results between covariance computation in Pandas and the MLLib equivalent. Results are reasonably close for fully specified inputs (i.e. without any NAs) ...
1
vote
1
answer
60
views
Convert RDD of Matrix to RDD of Vector
I have a RDD[Matrix[Double]] and want to convert it to RDD[Vector] (Each row in the Matrix will be converted to a Vector).
I've seen related answer like Convert Matrix to RowMatrix in Apache Spark ...
0
votes
0
answers
24
views
KolmogorovSmirnovTest spark
Hi I am trying to find data drift using the KolmogorovSmirnovTest function. What does the output of this function depicts. Pvalue and statistic. How can we check the data drift from it.any ...
0
votes
0
answers
27
views
py4j error o43.trainKMeansModel while calling KMeans.train in pyspark
I am trying to make clusters of user features extracted from a model trained using spark mllib ALS library following the steps stated in this paper.
The following code is used to load the saved model:
...
0
votes
0
answers
31
views
How to perform Principle Component Analysis on large feature set?
I am using logistic regression to train a dataset with 20,000 features and am also using PCA for dimensionality reduction. The snippet below is directly from the documentation here: https://spark....