79

I have the following sample DataFrame:

a    | b    | c   | 

1    | 2    | 4   |
0    | null | null| 
null | 3    | 4   |

And I want to replace null values only in the first 2 columns - Column "a" and "b":

a    | b    | c   | 

1    | 2    | 4   |
0    | 0    | null| 
0    | 3    | 4   |

Here is the code to create sample dataframe:

rdd = sc.parallelize([(1,2,4), (0,None,None), (None,3,4)])
df2 = sqlContext.createDataFrame(rdd, ["a", "b", "c"])

I know how to replace all null values using:

df2 = df2.fillna(0)

And when I try this, I lose the third column:

df2 = df2.select(df2.columns[0:1]).fillna(0)
1

2 Answers 2

149
df.fillna(0, subset=['a', 'b'])

There is a parameter named subset to choose the columns unless your spark version is lower than 1.3.1

2
  • wouldn't this give a new df with only one column? anyway to do this in place
    – Fizi
    Commented Mar 1, 2021 at 23:28
  • 2
    @Fizi in Spark, Dataframe is immutable, which means it's impossible to do this in place. Commented Nov 16, 2021 at 3:13
77

Use a dictionary to fill values of certain columns:

df.fillna( { 'a':0, 'b':0 } )
4
  • 1
    This is a better answer because it does not matter wether it is one or many values being filled in. Commented Jun 17, 2020 at 19:25
  • @ChrisMarotta Does the values type of all selected columns have to be of same type? Could it also be possible: df.fillna( { 'a':0, 'b':'2022-12-01' } ) where column a is of numeric type, and b is of date type?
    – nam
    Commented Jun 5, 2022 at 23:09
  • 2
    @nam, I suggest you fire up a pyspark terminal and find out Commented Jun 16, 2022 at 16:36
  • the behavious that @nam asked for is possible. see the third example in spark.apache.org/docs/3.1.3/api/python/reference/api/… Commented Jul 26, 2023 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.