Linked Questions

13 votes
1 answer
11k views

How to add my own function as a custom stage in a ML pyspark Pipeline? [duplicate]

The sample code from Florian -----------+-----------+-----------+ |ball_column|keep_the |hall_column| +-----------+-----------+-----------+ | 0| 7| 14| | 1| ...
PolarBear10's user avatar
  • 2,185
28 votes
6 answers
15k views

Serialize a custom transformer using python to be used within a Pyspark ML pipeline

I found the same discussion in comments section of Create a custom Transformer in PySpark ML, but there is no clear answer. There is also an unresolved JIRA corresponding to that: https://issues....
TechnoIndifferent's user avatar
24 votes
2 answers
12k views

How to create a custom Estimator in PySpark

I am trying to build a simple custom Estimator in PySpark MLlib. I have here that it is possible to write a custom Transformer but I am not sure how to do it on an Estimator. I also don't understand ...
Hanan Shteingart's user avatar
5 votes
2 answers
2k views

Transformer operating on multiple features in pyspark.ml

I want to make my own transformer of features in a DataFrame, so that I add a column which is, for example, a difference between two other columns. I followed this question, but the transformer there ...
aliva7's user avatar
  • 211
5 votes
2 answers
5k views

How can I declare a Column as a categorical feature in a DataFrame for use in ml

How can I declare that a given Column in my DataFrame contains categorical information? I have a Spark SQL DataFrame which I loaded from a database. Many of the columns in this DataFrame have ...
unda91's user avatar
  • 63
3 votes
1 answer
5k views

How to create a custom tokenizer in PySpark ML

sentenceDataFrame = spark.createDataFrame([ (0, "Hi I heard about Spark"), (1, "I wish Java could use case classes"), (2, "Logistic,regression,models,are,neat") ], ["id", "...
ShuoshuoFan's user avatar
3 votes
1 answer
3k views

How to add UDF in pyspark pipeline?

I have the following code which basically is doing feature engineering pipeline: token_q1=Tokenizer(inputCol='question1',outputCol='question1_tokens') token_q2=Tokenizer(inputCol='question2',...
CyberPunk's user avatar
  • 1,347
7 votes
0 answers
3k views

Custom Transformer in PySpark Pipeline with Cross Validation

I wrote a custom transformer like it is described here. When creating a pipeline with my transformer as first step I am able to train a (Logistic Regression) model for classification. However, when ...
vkoe's user avatar
  • 381
5 votes
1 answer
2k views

How can I save partial results of dataframe transformation processes in pyspark?

I am working in apache-spark to make multiple transformations on a single Dataframe with python. I have coded some functions in order to make easier the different transformations. Imagine we have ...
Hugo Reyes's user avatar
  • 1,585
3 votes
1 answer
2k views

Create a Python transformer on sparsevector data type column in Pyspark ML

I have a dataframe with a column 'features'(each row in the dataframe represents a document). I used HashingTF to calculate the column 'tf' and I also created a custom transformer 'TermCount' (just as ...
K.Ali's user avatar
  • 283
0 votes
1 answer
417 views

How to make a method JSON serializable to be used in a Custom Pyspark Transformer

I am using pyspark(2.3.0) api to create a custom transformer. I want to create a simple transformer which takes any function as a param. I tried doing it with using identity in TypeConverters. The ...
Heisenbug's user avatar
  • 126
0 votes
1 answer
232 views

Best way to Create a custom Transformer In Java spark ml

I am learning Big data using Apache spark and I want to create a custom transformer for Spark ml so that I can execute some aggregate functions or can perform other possible operation on it
ngi's user avatar
  • 51
0 votes
1 answer
116 views

Why PySpark execute only the default statement in my custom `SQLTransformer`

I wrote a custom SQLTransformer in PySpark. And setting a default SQL statement is mandatory to have the code being executed. I can save the custum transformer within Python, load it and execute it ...
Bentech's user avatar
  • 478