41

I have looked up this issue and most questions are for more complex replacements. However in my case I have a very simple dataframe as a test dummy.

The aim is to replace a string anywhere in the dataframe with an nan, however this does not seem to work (i.e. does not replace; no errors whatsoever). I've tried replacing with another string and it does not work either. E.g.

d = {'color' : pd.Series(['white', 'blue', 'orange']),
   'second_color': pd.Series(['white', 'black', 'blue']),
   'value' : pd.Series([1., 2., 3.])}
df = pd.DataFrame(d)
df.replace('white', np.nan)

The output is still:

      color second_color  value
  0   white        white      1
  1    blue        black      2
  2  orange         blue      3

This problem is often addressed using inplace=True, but there are caveats to that. Please also see Understanding inplace=True in pandas.

9 Answers 9

149

Given that this is the top Google result when searching for "Pandas replace is not working" I'd like to also mention that:

replace does full replacement searches, unless you turn on the regex switch. Use regex=True, and it should perform partial replacements as well.

This took me 30 minutes to find out, so hopefully I've saved the next person 30 minutes.

1
  • 4
    Exactly what I was bumping up against for a replace failing within a delimited string.
    – Will B
    Aug 28, 2018 at 14:39
40

You need to assign back

df = df.replace('white', np.nan)

or pass param inplace=True:

In [50]:
d = {'color' : pd.Series(['white', 'blue', 'orange']),
   'second_color': pd.Series(['white', 'black', 'blue']),
   'value' : pd.Series([1., 2., 3.])}
df = pd.DataFrame(d)
df.replace('white', np.nan, inplace=True)
df

Out[50]:
    color second_color  value
0     NaN          NaN    1.0
1    blue        black    2.0
2  orange         blue    3.0

Most pandas ops return a copy and most have param inplace which is usually defaulted to False

5
  • 11
    Note that there are real bugs that result in replace really not working in some cases (see issue #29813).
    – bluenote10
    Nov 23, 2019 at 7:54
  • 2
    I just ran into an issue where df.replace(x,y, inplace=True) did not work but df = df.replace(x,y) did work. x was an int64 and y was a float64
    – MattR
    Dec 30, 2019 at 19:40
  • same with @MattR. inplace not working for some reason
    – greendino
    Jan 14, 2021 at 0:54
  • Ah - I had similar issues where inplace=True had apparenty no effect, while assigning to a new variable showed that the replacement actually worked. I was using a regex to remove whitespaces & suchs. Nov 23, 2021 at 21:27
  • I believe df.foo(x, y, inplace=True) is now deprecated, in favor of df = df.foo(x, y)
    – jazcap53
    Jan 13, 2022 at 13:38
14

Neither one with inplace=True nor the other with regex=True don't work in my case. So I found a solution with using Series.str.replace instead. It can be useful if you need to replace a substring.

In [4]: df['color'] = df.color.str.replace('e', 'E!')
In [5]: df  
Out[5]: 
     color second_color  value
0   whitE!        white    1.0
1    bluE!        black    2.0
2  orangE!         blue    3.0

or even with a slicing.

In [10]: df.loc[df.color=='blue', 'color'] = df.color.str.replace('e', 'E!')
In [11]: df  
Out[11]: 
    color second_color  value
0   white        white    1.0
1   bluE!        black    2.0
2  orange         blue    3.0
12

You might need to check the data type of the column before using replace function directly. It could be the case that you are using replace function on Object data type, in this case, you need to apply replace function after converting it into a string.

Wrong:

df["column-name"] = df["column-name"].replace('abc', 'def')

Correct:

df["column-name"] = df["column-name"].str.replace('abc', 'def')
7

When you use df.replace() it creates a new temporary object, but doesn't modify yours. You can use one of the two following lines to modify df:

df = df.replace('white', np.nan)
df.replace('white', np.nan, inplace = True)
2

What worked for me was using this dict notation.

{old_value:new_value}

df.replace({10:100},inplace=True)

check the documentation for more info. https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.replace.html

0
df.replace({'white': np.nan}, inplace=True, regex=True)
0

Python 3.10, pandas 1.4.2, inplace=True did not work for below example (column dtype int32), but reassigning it did.

df["col"].replace[[0, 130], [12555555, 12555555], inplace=True)  # NOT work
df["col"] = df["col"].replace[[0, 130], [12555555, 12555555])   # worked

... and in another situation involving nans in text columns, the column needed typing in a pre-step (not just .str, as above):

df["col"].replace[["man", "woman", np.nan], [1, 2, -1], inplace=True)  # NOT work
df["col"] = df["col"].str.replace[["man", "woman", np.nan], [1, 2, -1])     # NOT work

df["col"] = df["col"].astype(str)    # needed
df["col"] = df["col"].replace[["man", "woman", np.nan], [1, 2, -1])   # worked
-1

One other reason, where i faced .replace function was not working and i found the reason and fixed.

If you have the string in the column as "word1 word2", when read from excel, the space in between "word1" and "word2" has the "nbsp" meaning non blank spacing. If we replace with normal space, everything works fine. My column name is "Name"

    nonBreakSpace = u'\xa0'
    df['Name'] = df['Name'].replace(nonBreakSpace,' ',regex=True)
    df['Name']=df["Name"].str.replace("replace with","replace to",regex=True)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.