11

Inputs:

df['PopEst']
    .astype('float')
    .groupby(ContinentDict)
    .agg(['size','sum','mean','std']))

Outputs:

            size            sum                mean              std
Asia          5     2.898666e+09       5.797333e+08     6.790979e+08
Australia     1     2.331602e+07       2.331602e+07              NaN
Europe        6     4.579297e+08       7.632161e+07     3.464767e+07
North America 2     3.528552e+08       1.764276e+08     1.996696e+08
South America 1     2.059153e+08       2.059153e+08              NaN

Some values in column of std turns out to be NaN if the group just have one row, but I think these values are supposed to be 0, why is that?

0

2 Answers 2

14

pd.DataFrame.std assumes 1 degree of freedom by default, also known as sample standard deviation. This results in NaN results for groups with one number.

numpy.std, by contrast, assumes 0 degree of freedom by default, also known as population standard deviation. This gives 0 for groups with one number.

To understand the difference between sample and population, see Bessel's correction.

Therefore, you can specify numpy.std for your calculation. Note, however, that the output will be different as the calculation is different. Here's a minimal example.

import pandas as pd, numpy as np

df = pd.DataFrame(np.random.randint(0, 9, (5, 2)))

def std(x): return np.std(x)

res = df.groupby(0)[1].agg(['size', 'sum', 'mean', std])

print(res)

   size  sum  mean       std
0                           
0     2   13   6.5       0.5
4     1    3   3.0       0.0
5     1    3   3.0       0.0
6     1    3   3.0       0.0

Alternatively, if you require 1 degree of freedom, you can use fillna to replace NaN values with 0:

res = df.groupby(0)[1].agg(['size', 'sum', 'mean', 'std']).fillna(0)
7
  • 3
    Thanks for your answer! When I change the code to .agg('size','sum','mean',np.std), the outputs still keep NaN. But when I use lambda x:np.std(x) instead, NaNs turn to 0, it is what I wanted. I wonder know why this happened.
    – Alex J
    Commented May 13, 2018 at 7:27
  • what if I want to be able to change ddof when calling .agg([np.mean, np.std])? I tried to pass it as external argument with .agg([np.mean, np.std], ddof=0) but it doesn't work. Commented May 1, 2020 at 6:38
  • @FrancescoLS, As in my answer, use a function, e.g. def std_zero(x): return np.std(x, ddof=0)
    – jpp
    Commented May 1, 2020 at 7:08
  • @jpp yes, I see the point and it works. But what I am saying is that I want to be able to pass it as an argument, e.g. say I want to compare the std with ddof =1 and =2, doing as you suggest I would need to create 2 different functions and pass them Commented May 1, 2020 at 13:57
  • 1
    @FrancescoLS, The best I can think of.. you can use a higher order function via functools.partial, then use partial(np.std, ddof=1) or partial(np.std, ddof=2) etc.
    – jpp
    Commented May 1, 2020 at 16:16
0

According to the document, np.std(..., ddof=1) by default set "delta degree of freedom" to 1. To fix your problem, simply replace np.std with lambda x: np.std(x, ddof=0) then your NaN will be changed to 0.

1
  • This answer seems incorrect.. the document linked refers to pd.DataFrame.std while the answer specifies it relates to np.std.
    – jpp
    Commented Oct 30, 2019 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.