stats.zscore
from scipy
stats.zscore
(which was mentioned in Manuel's answer) works on DataFrames / 2D arrays, so it's not necessary to call it via apply()
(because apply
is a syntactic sugar of a Python for-loop, if there are a lot of columns, it will be noticeably slow1). Syntactically, very minimal change is required as well; simply call zscore
on the DataFrame.
from scipy import stats
df = pd.DataFrame([[0,1,2],[3,3,5],[5,6,100]]).add_prefix('col')
zscore_df = stats.zscore(df)
If certain columns need to be normalized, simply select those columns and compute z-score.
stats.zscore(df[['col0', 'col2']])
You can verify that this does indeed return the same DataFrame as applying zscore
on each column and manual computation ((df - df.mean())/df.std(ddof=0)
).
x = stats.zscore(df)
y = df.apply(stats.zscore)
z = (df - df.mean()) / df.std(ddof=0)
np.allclose(x, y) and np.allclose(x, z) # True
StandardScaler
from scikit-learn
Another way is to call StandardScaler()
from scikit-learn. Simply instantiate StandardScaler
and call fit_transform
using the relevant columns as input. The result is a numpy array which you can assign back to the dataframe as new columns (or work on the array itself etc.).
from sklearn.preprocessing import StandardScaler
cols = ['col1', 'col2']
new_cols = [f"{c}_zscore" for c in cols]
sc = StandardScaler()
df[new_cols] = sc.fit_transform(df[cols])
1 A timeit test shows that for a DataFrame with 100 columns, calling zscore
directly on the columns is ~30 times faster than calling it on each column using apply()
. Also, direct computation as mentioned in Joe Bathelt's answer actually performs the best.
import pandas as pd
import numpy as np
from scipy import stats
from sklearn.preprocessing import StandardScaler
df = pd.DataFrame(np.random.default_rng(0).choice(100, size=(1000, 100))).add_prefix('col')
%timeit df.apply(stats.zscore)
# 105 ms ± 3.25 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit stats.zscore(df)
# 3.63 ms ± 209 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit df.sub(df.mean()).div(df.std(ddof=0))
# 2.86 ms ± 208 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit StandardScaler().fit_transform(df)
# 6.89 ms ± 235 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)