26

I face a problem of modification of a dataframe inside a function that I have never observed previously. Is there a method to deal with this so that the initial dataframe is not modified.

In[30]: def test(df):
    df['tt'] = np.nan
    return df

In[31]: dff = pd.DataFrame(data=[])

In[32]: dff

Out[32]: 
Empty DataFrame
Columns: []
Index: []
In[33]: df = test(dff)

In[34]: dff

Out[34]: 
Empty DataFrame
Columns: [tt]
Index: []
7
  • 4
    Pass a copy of the dataframe? Or make one inside the function, and mutate and return that? It's bad form to mutate an argument and return anything other than None.
    – jonrsharpe
    Jul 24, 2015 at 15:09
  • It's a solution but not memory efficient. But it's the first time I face that. Due to the version 0.16.2 ?
    – Alexis G
    Jul 24, 2015 at 15:10
  • 1
    you can call .copy() to take an explicit deep copy
    – EdChum
    Jul 24, 2015 at 15:10
  • 1
    Nope, nothing to do with changing versions - this behaviour is the same for all mutable objects passed to Python functions, unique neither to Pandas generally nor v0.16.2 specifically.
    – jonrsharpe
    Jul 24, 2015 at 15:11
  • Can you tell us a bit more about your use case? If you want to return the df at the end of the function, I don't think you can avoid doing a .copy()
    – cd98
    Jul 24, 2015 at 22:27

1 Answer 1

59
def test(df):
    df = df.copy(deep=True)
    df['tt'] = np.nan
    return df

If you pass the dataframe into a function and manipulate it and return the same dataframe, you are going to get the same dataframe in modified version. If you want to keep your old dataframe and create a new dataframe with your modifications then by definition you have to have 2 dataframes. The one that you pass in that you don't want modified and the new one that is modified. Therefore, if you don't want to change the original dataframe your best bet is to make a copy of the original dataframe. In my example I rebound the variable "df" in the function to the new copied dataframe. I used the copy method and the argument "deep=True" makes a copy of the dataframe and its contents. You can read more here:http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.copy.html

2
  • Is this also true for pyspark dataframes?
    – nigelhenry
    May 13, 2021 at 14:53
  • 2
    Thanks! I've been using pandas for a while and just came across this myself just now. training a model on a dataframe and inside training function it make some changes to df but does not return it. This still leads to modification of original dataframe. Copying is the only way?
    – haneulkim
    Nov 1, 2022 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.