80

I have a dataframe containing a single column of IDs and all other columns are numerical values for which I want to compute z-scores. Here's a subsection of it:

ID      Age    BMI    Risk Factor
PT 6    48     19.3    4
PT 8    43     20.9    NaN
PT 2    39     18.1    3
PT 9    41     19.5    NaN

Some of my columns contain NaN values which I do not want to include into the z-score calculations so I intend to use a solution offered to this question: how to zscore normalize pandas column with nans?

df['zscore'] = (df.a - df.a.mean())/df.a.std(ddof=0)

I'm interested in applying this solution to all of my columns except the ID column to produce a new dataframe which I can save as an Excel file using

df2.to_excel("Z-Scores.xlsx")

So basically; how can I compute z-scores for each column (ignoring NaN values) and push everything into a new dataframe?

SIDENOTE: there is a concept in pandas called "indexing" which intimidates me because I do not understand it well. If indexing is a crucial part of solving this problem, please dumb down your explanation of indexing.

9 Answers 9

109

Using Scipy's zscore function:

df = pd.DataFrame(np.random.randint(100, 200, size=(5, 3)), columns=['A', 'B', 'C'])
df

|    |   A |   B |   C |
|---:|----:|----:|----:|
|  0 | 163 | 163 | 159 |
|  1 | 120 | 153 | 181 |
|  2 | 130 | 199 | 108 |
|  3 | 108 | 188 | 157 |
|  4 | 109 | 171 | 119 |

from scipy.stats import zscore
df.apply(zscore)

|    |         A |         B |         C |
|---:|----------:|----------:|----------:|
|  0 |  1.83447  | -0.708023 |  0.523362 |
|  1 | -0.297482 | -1.30804  |  1.3342   |
|  2 |  0.198321 |  1.45205  | -1.35632  |
|  3 | -0.892446 |  0.792025 |  0.449649 |
|  4 | -0.842866 | -0.228007 | -0.950897 |

If not all the columns of your data frame are numeric, then you can apply the Z-score function only to the numeric columns using the select_dtypes function:

# Note that `select_dtypes` returns a data frame. We are selecting only the columns
numeric_cols = df.select_dtypes(include=[np.number]).columns
df[numeric_cols].apply(zscore)

|    |         A |         B |         C |
|---:|----------:|----------:|----------:|
|  0 |  1.83447  | -0.708023 |  0.523362 |
|  1 | -0.297482 | -1.30804  |  1.3342   |
|  2 |  0.198321 |  1.45205  | -1.35632  |
|  3 | -0.892446 |  0.792025 |  0.449649 |
|  4 | -0.842866 | -0.228007 | -0.950897 |
3
  • How can I compute a rolling z-score?
    – Nermin
    Commented Apr 7, 2023 at 13:13
  • @nermin, Someone else asked the same before in StackOverflow. You can find the answers that other people gave if you search for the question.
    – Manuel
    Commented Apr 11, 2023 at 17:33
  • I'm getting numpy.exceptions.AxisError: axis 0 is out of bounds for array of dimension 0 - when applying it to single column - any ideas how to solve it? Commented Nov 9, 2023 at 12:17
94

Build a list from the columns and remove the column you don't want to calculate the Z score for:

In [66]:
cols = list(df.columns)
cols.remove('ID')
df[cols]

Out[66]:
   Age  BMI  Risk  Factor
0    6   48  19.3       4
1    8   43  20.9     NaN
2    2   39  18.1       3
3    9   41  19.5     NaN
In [68]:
# now iterate over the remaining columns and create a new zscore column
for col in cols:
    col_zscore = col + '_zscore'
    df[col_zscore] = (df[col] - df[col].mean())/df[col].std(ddof=0)
df
Out[68]:
   ID  Age  BMI  Risk  Factor  Age_zscore  BMI_zscore  Risk_zscore  \
0  PT    6   48  19.3       4   -0.093250    1.569614    -0.150946   
1  PT    8   43  20.9     NaN    0.652753    0.074744     1.459148   
2  PT    2   39  18.1       3   -1.585258   -1.121153    -1.358517   
3  PT    9   41  19.5     NaN    1.025755   -0.523205     0.050315   

   Factor_zscore  
0              1  
1            NaN  
2             -1  
3            NaN  
30

If you want to calculate the zscore for all of the columns, you can just use the following:

df_zscore = (df - df.mean())/df.std()
1
  • 6
    @pitosalas: @ascripter, you are correct. Passing df.std(ddof=0) produces the same result as df.apply(scipy.stats.zscore)
    – roob
    Commented Sep 26, 2018 at 21:24
8

Here's other way of getting Zscore using custom function:

In [6]: import pandas as pd; import numpy as np

In [7]: np.random.seed(0) # Fixes the random seed

In [8]: df = pd.DataFrame(np.random.randn(5,3), columns=["randomA", "randomB","randomC"])

In [9]: df # watch output of dataframe
Out[9]:
    randomA   randomB   randomC
0  1.764052  0.400157  0.978738
1  2.240893  1.867558 -0.977278
2  0.950088 -0.151357 -0.103219
3  0.410599  0.144044  1.454274
4  0.761038  0.121675  0.443863

## Create custom function to compute Zscore 
In [10]: def z_score(df):
   ....:         df.columns = [x + "_zscore" for x in df.columns.tolist()]
   ....:         return ((df - df.mean())/df.std(ddof=0))
   ....:

## make sure you filter or select columns of interest before passing dataframe to function
In [11]: z_score(df) # compute Zscore
Out[11]:
   randomA_zscore  randomB_zscore  randomC_zscore
0        0.798350       -0.106335        0.731041
1        1.505002        1.939828       -1.577295
2       -0.407899       -0.875374       -0.545799
3       -1.207392       -0.463464        1.292230
4       -0.688061       -0.494655        0.099824

Result reproduced using scipy.stats zscore

In [12]: from scipy.stats import zscore

In [13]: df.apply(zscore) # (Credit: Manuel)
Out[13]:
    randomA   randomB   randomC
0  0.798350 -0.106335  0.731041
1  1.505002  1.939828 -1.577295
2 -0.407899 -0.875374 -0.545799
3 -1.207392 -0.463464  1.292230
4 -0.688061 -0.494655  0.099824
6

for Z score, we can stick to documentation instead of using 'apply' function

from scipy.stats import zscore
df_zscore = zscore(cols as array, axis=1)
1
  • 1
    what is cols? (the original dataframe?) this answer would benefit from more context Commented Mar 25 at 20:42
4

The almost one-liner solution:

df2 = (df.ix[:,1:] - df.ix[:,1:].mean()) / df.ix[:,1:].std()
df2['ID'] = df['ID']
0
2

stats.zscore from scipy

stats.zscore (which was mentioned in Manuel's answer) works on DataFrames / 2D arrays, so it's not necessary to call it via apply() (because apply is a syntactic sugar of a Python for-loop, if there are a lot of columns, it will be noticeably slow1). Syntactically, very minimal change is required as well; simply call zscore on the DataFrame.

from scipy import stats
df = pd.DataFrame([[0,1,2],[3,3,5],[5,6,100]]).add_prefix('col')
zscore_df = stats.zscore(df)

result

If certain columns need to be normalized, simply select those columns and compute z-score.

stats.zscore(df[['col0', 'col2']])

You can verify that this does indeed return the same DataFrame as applying zscore on each column and manual computation ((df - df.mean())/df.std(ddof=0)).

x = stats.zscore(df)
y = df.apply(stats.zscore)
z = (df - df.mean()) / df.std(ddof=0)
np.allclose(x, y) and np.allclose(x, z)  # True

StandardScaler from scikit-learn

Another way is to call StandardScaler() from scikit-learn. Simply instantiate StandardScaler and call fit_transform using the relevant columns as input. The result is a numpy array which you can assign back to the dataframe as new columns (or work on the array itself etc.).

from sklearn.preprocessing import StandardScaler

cols = ['col1', 'col2']
new_cols = [f"{c}_zscore" for c in cols]

sc = StandardScaler()
df[new_cols] = sc.fit_transform(df[cols])

1 A timeit test shows that for a DataFrame with 100 columns, calling zscore directly on the columns is ~30 times faster than calling it on each column using apply(). Also, direct computation as mentioned in Joe Bathelt's answer actually performs the best.

import pandas as pd
import numpy as np
from scipy import stats
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(np.random.default_rng(0).choice(100, size=(1000, 100))).add_prefix('col')

%timeit df.apply(stats.zscore)
# 105 ms ± 3.25 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit stats.zscore(df)
# 3.63 ms ± 209 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit df.sub(df.mean()).div(df.std(ddof=0))
# 2.86 ms ± 208 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit StandardScaler().fit_transform(df)
# 6.89 ms ± 235 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
1

When we are dealing with time-series, calculating z-scores (or anomalies - not the same thing, but you can adapt this code easily) is a bit more complicated. For example, you have 10 years of temperature data measured weekly. To calculate z-scores for the whole time-series, you have to know the means and standard deviations for each day of the year. So, let's get started:

Assume you have a pandas DataFrame. First of all, you need a DateTime index. If you don't have it yet, but luckily you do have a column with dates, just make it as your index. Pandas will try to guess the date format. The goal here is to have DateTimeIndex. You can check it out by trying:

type(df.index)

If you don't have one, let's make it.

df.index = pd.DatetimeIndex(df[datecolumn])
df = df.drop(datecolumn,axis=1)

Next step is to calculate mean and standard deviation for each group of days. For this, we use the groupby method.

mean = pd.groupby(df,by=[df.index.dayofyear]).aggregate(np.nanmean)
std = pd.groupby(df,by=[df.index.dayofyear]).aggregate(np.nanstd)

Finally, we loop through all the dates, performing the calculation (value - mean)/stddev; however, as mentioned, for time-series this is not so straightforward.

df2 = df.copy() #keep a copy for future comparisons 
for y in np.unique(df.index.year):
    for d in np.unique(df.index.dayofyear):
        df2[(df.index.year==y) & (df.index.dayofyear==d)] = (df[(df.index.year==y) & (df.index.dayofyear==d)]- mean.ix[d])/std.ix[d]
        df2.index.name = 'date' #this is just to look nicer

df2 #this is your z-score dataset.

The logic inside the for loops is: for a given year we have to match each dayofyear to its mean and stdev. We run this for all the years in your time-series.

0

To calculate a z-score for an entire column quickly, do as follows:

from scipy.stats import zscore
import pandas as pd

df = pd.DataFrame({'num_1': [1,2,3,4,5,6,7,8,9,3,4,6,5,7,3,2,9]})
df['num_1_zscore'] = zscore(df['num_1'])

display(df)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.