I would like to rewrite this from R to Pyspark, any nice looking suggestions?
array <- c(1,2,3)
dataset <- filter(!(column %in% array))
In pyspark you can do it like this:
array = [1, 2, 3]
dataframe.filter(dataframe.column.isin(array) == False)
Or using the binary NOT operator:
dataframe.filter(~dataframe.column.isin(array))
dataframe.column
is case sensitive! Alternatively, you can use the dictionary syntax dataframe[column]
, which is not :)
Commented
Aug 14, 2019 at 20:44
==
operator is doing here is calling the overloaded __eq__
method on the Column
result returned by dataframe.column.isin(*array)
. That's overloaded to return another column result to test for equality with the other argument (in this case, False
). The is
operator tests for object identity, that is, if the objects are actually the same place in memory. If you use is
here, it would always fail because the constant False
doesn't ever live in the same memory location as a Column
. Additionally, you can't overload is
.
Commented
Sep 27, 2019 at 21:23
isin(array)
and it works just fine.
Commented
Aug 4, 2020 at 10:38
Take the operator ~ which means contrary :
df_filtered = df.filter(~df["column_name"].isin([1, 2, 3]))
== False
when we have ~
specifically for negation?
Commented
Oct 9, 2019 at 12:50
slightly different syntax and a "date" data set:
toGetDates={'2017-11-09', '2017-11-11', '2017-11-12'}
df= df.filter(df['DATE'].isin(toGetDates) == False)
You can use the .subtract()
buddy.
Example:
df1 = df.select(col(1),col(2),col(3))
df2 = df.subtract(df1)
This way, df2 will be defined as everything that is df that is not df1.
*
is not needed. So:
list = [1, 2, 3]
dataframe.filter(~dataframe.column.isin(list))
You can also use sql functions .col
+ .isin()
:
import pyspark.sql.functions as F
array = [1,2,3]
df = df.filter(~F.col(column_name).isin(array))
This might be useful if you are using sql functions and want consistency.
You can also loop the array and filter:
array = [1, 2, 3]
for i in array:
df = df.filter(df["column"] != i)