293

I am trying to fill none values in a Pandas dataframe with 0's for only some subset of columns.

When I do:

import pandas as pd
df = pd.DataFrame(data={'a':[1,2,3,None],'b':[4,5,None,6],'c':[None,None,7,8]})
print df
df.fillna(value=0, inplace=True)
print df

The output:

     a    b    c
0  1.0  4.0  NaN
1  2.0  5.0  NaN
2  3.0  NaN  7.0
3  NaN  6.0  8.0
     a    b    c
0  1.0  4.0  0.0
1  2.0  5.0  0.0
2  3.0  0.0  7.0
3  0.0  6.0  8.0

It replaces every None with 0's. What I want to do is, only replace Nones in columns a and b, but not c.

What is the best way of doing this?

1
  • Would be great if more pandas methods having inplace also had include and exclude (subset of columns). And regex for that matter.
    – BSalita
    Commented Oct 14, 2023 at 20:56

10 Answers 10

429

You can select your desired columns and do it by assignment:

df[['a', 'b']] = df[['a','b']].fillna(value=0)

The resulting output is as expected:

     a    b    c
0  1.0  4.0  NaN
1  2.0  5.0  NaN
2  3.0  0.0  7.0
3  0.0  6.0  8.0
9
  • 3
    Yes, this is exactly what I want! Thank you. Any ways to do this inplace? My original dataframe is pretty big.
    – Sait
    Commented Jun 30, 2016 at 22:10
  • 3
    I don't think there is any performance gain by doing this in place as you're overwriting the orig df anyway
    – EdChum
    Commented Jun 30, 2016 at 22:12
  • 7
    The loc is superfluous here, df[['a', 'b']] = df[['a','b']].fillna(value=0) will still work
    – EdChum
    Commented Jun 30, 2016 at 22:13
  • 3
    @EdChum Doesn't it produce a temporary data frame and hence need more memory to do so? (I am concerned more about memory than time complexity.)
    – Sait
    Commented Jun 30, 2016 at 22:14
  • 11
    For many operations, inplace will still work on a copy. I don't know if it's the case for fillna or not. See this answer from one of the pandas core developers.
    – root
    Commented Jun 30, 2016 at 22:16
210

You can using dict , fillna with different value for different column

df.fillna({'a':0,'b':0})
Out[829]: 
     a    b    c
0  1.0  4.0  NaN
1  2.0  5.0  NaN
2  3.0  0.0  7.0
3  0.0  6.0  8.0

After assign it back

df=df.fillna({'a':0,'b':0})
df
Out[831]: 
     a    b    c
0  1.0  4.0  NaN
1  2.0  5.0  NaN
2  3.0  0.0  7.0
3  0.0  6.0  8.0
5
  • 2
    really cool, Btw for the dict you can use fromkeys if you want, +1 Commented Aug 29, 2018 at 1:14
  • 1
    The answer/example would be clearer if it actually showed different values for the different columns.
    – RufusVS
    Commented Sep 21, 2018 at 18:00
  • @RufusVS that is right , but still try to match the op's expected output
    – BENY
    Commented Sep 21, 2018 at 18:04
  • 5
    This is the better solution that the accepted answer, because it avoids chained indexing issues, e.g. if used with df.fillna({'a':0,'b':0}, inplace=True)
    – Alex
    Commented Apr 6, 2020 at 10:39
  • 3
    How to use methods like ffill or bfill inside a dictionary? Commented Dec 28, 2021 at 18:44
58

You can avoid making a copy of the object using Wen's solution and inplace=True:

df.fillna({'a':0, 'b':0}, inplace=True)
print(df)

Which yields:

     a    b    c
0  1.0  4.0  NaN
1  2.0  5.0  NaN
2  3.0  0.0  7.0
3  0.0  6.0  8.0
1
19

using the top answer produces a warning about making changes to a copy of a df slice. Assuming that you have other columns, a better way to do this is to pass a dictionary:
df.fillna({'A': 'NA', 'B': 'NA'}, inplace=True)

12

This should work and without copywarning

df[['a', 'b']] = df.loc[:,['a', 'b']].fillna(value=0)
0
9

Here's how you can do it all in one line:

df[['a', 'b']].fillna(value=0, inplace=True)

Breakdown: df[['a', 'b']] selects the columns you want to fill NaN values for, value=0 tells it to fill NaNs with zero, and inplace=True will make the changes permanent, without having to make a copy of the object.

2
  • 6
    Somehow this gives SettingWithCopyWarning and the change is not reflected in df.
    – Michael
    Commented Nov 4, 2020 at 15:18
  • Is there a way to fillna(0) on every column to the right of the nth column? Commented Jan 6, 2023 at 14:52
6

For some odd reason this DID NOT work (using Pandas: '0.25.1')

df[['col1', 'col2']].fillna(value=0, inplace=True)

Another solution:

subset_cols = ['col1','col2']
[df[col].fillna(0, inplace=True) for col in subset_cols]

Example:

df = pd.DataFrame(data={'col1':[1,2,np.nan,], 'col2':[1,np.nan,3], 'col3':[np.nan,2,3]})

output:

   col1  col2  col3
0  1.00  1.00   nan
1  2.00   nan  2.00
2   nan  3.00  3.00

Apply list comp. to fillna values:

subset_cols = ['col1','col2']
[df[col].fillna(0, inplace=True) for col in subset_cols]

Output:

   col1  col2  col3
0  1.00  1.00   nan
1  2.00  0.00  2.00
2  0.00  3.00  3.00
3
  • I think inplace is not good practice, check this and this
    – jezrael
    Commented Dec 4, 2020 at 6:55
  • So the best should be if raise inplace warning and then removed from pandas in my opinion.
    – jezrael
    Commented Dec 4, 2020 at 6:56
  • So easy advice - always avoid inplace and never such problem here ;)
    – jezrael
    Commented Dec 4, 2020 at 6:57
5

Or something like:

df.loc[df['a'].isnull(),'a']=0
df.loc[df['b'].isnull(),'b']=0

and if there is more:

for i in your_list:
    df.loc[df[i].isnull(),i]=0
1

Sometimes this syntax wont work:

df[['col1','col2']] = df[['col1','col2']].fillna()

Use the following instead:

df['col1','col2']
0

If you're looking for a more efficient way,

for col in ['a', 'b']:
    v = df.loc[:, col].values
    np.nan_to_num(v, 0.0)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.