5

For certain Pandas functions, such as sum(), cumsum() and cumprod(), there is an option for skipna which is set to True by default. This causes issues for me as errors might silently propagate so I always explicitly set skipna to False.

sum_df = df.sum(skipna=False)

Doing it every time one of these functions appear makes the code look a bit unwieldy. Is there a way I can change the default behaviour in Pandas?

2
  • Have you found a solution to your problem?
    – jlandercy
    May 15, 2019 at 17:20
  • Not a satisfactory one.
    – Spinor8
    May 16, 2019 at 4:32

2 Answers 2

4

Option is not an option (yet)

It seems there is nothing such an option to control this behaviour. It is hard coded:

import inspect
inspect.getfile(pd.DataFrame.sum)    # './pandas/core/generic.py'
inspect.getsource(pd.DataFrame.sum)

# @Substitution(outname=name, desc=desc, name1=name1, name2=name2,
#                  axis_descr=axis_descr, min_count=_min_count_stub,
#                  see_also=see_also, examples=examples)
# @Appender(_num_doc)
# def stat_func(self, axis=None, skipna=None, level=None, numeric_only=None,
# [...]

It could be a good idea for pull request.

A simple solution

Probably not the best solution, it is a bit hackish but it does address your problem.

I am not saying that it is a good practice in general. It may have drawbacks that I have not addressed (you are welcome to list it in comment). Anyway this solution has the advantage to be non intrusive.

Additionally, although it is a quite simple technique and it is pure PSL, it may violate Principle Of Least Astonishment (see this answer for details).

MCVE

Lets build a wrapper that overrides existing default parameters or add extra parameters:

def set_default(func, **default):
    def inner(*args, **kwargs):
        kwargs.update(default)        # Update function kwargs w/ decorator defaults
        return func(*args, **kwargs)  # Call function w/ updated kwargs
    return inner                      # Return decorated function

Then, we can decorate any function. For instance:

import pandas as pd
pd.DataFrame.sum = set_default(pd.DataFrame.sum, skipna=False)

Then, the sum method of DataFrame object has its skipna overridden to False each time we call it. Now the following code:

import numpy as np
df = pd.DataFrame([1., 2., np.nan])
df.sum()

Returns:

0   NaN
dtype: float64

Instead of:

0    3.0
dtype: float64

Automation

We can apply this modification to many functions, at once:

for key in ['sum', 'mean', 'std']:
    setattr(pd.DataFrame, key, set_default(getattr(pd.DataFrame, key), skipna=False))

If we store those modifications into a python module (.py file) they will be applied at the import time without having the need to modify the Pandas code itself.

0

It it perhaps not the best way, but I think you should modify the pandas’ file.

1. Find the pandas’ file.

If you installed pandas with pip, it should be in the Lib file, which is located in your python installation folder. If you do not know where python is installed, look it up on Google.

If you didn’t installed it with pip, look up on Google where are located packages installed with your software (anaconda or other)

2. Find the DataFrame class and the method.

The best way to do that, is to do CTRL+F or other letter to look for the method

3. Change the default value.

When you found the method(s), change skipna=True to skipna=False.

After you’ve done this, save the file and it should work !

I hope I helped you :)

1
  • Pandas sum method is not located where DataFrame object resides and it is substitution of stat_func. IMO, the best way to find where the code actually is without having to resort to google or find a keyword that is not present in a 10k lines file is inspect module which provides the required information in one single call. Changing, improving code is a good idea, sharing with other and test it is even better. Keeping track of modification is essential. Why not fork the package or make a pull request?
    – jlandercy
    May 17, 2019 at 7:04

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.