I can compare two Pandas series for exact equality using pandas.Series.equals
. Is there a corresponding function or parameter that will check if the elements are equal to some ε of precision?
4 Answers
You can use numpy.allclose
:
numpy.allclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False)
Returns
True
if two arrays are element-wise equal within a tolerance.The tolerance values are positive, typically very small numbers. The relative difference (
rtol * abs(b)
) and the absolute differenceatol
are added together to compare against the absolute difference betweena
andb
.
numpy
works well with pandas.Series
objects, so if you have two of them - s1
and s2
, you can simply do:
np.allclose(s1, s2, atol=...)
Where atol
is your tolerance value.
Numpy works well with pandas Series. However one has to be careful with the order of indices (or columns and indices for pandas DataFrame)
For example
series_1 = pd.Series(data=[0,1], index=['a','b'])
series_2 = pd.Series(data=[1,0], index=['b','a'])
np.allclose(series_1,series_2)
will return False
A workaround is to use the index of one pandas series
np.allclose(series_1, series_2.loc[series_1.index])
If you want to avoid numpy, there is another way, use assert_series_equal
import pandas as pd
s1 = pd.Series([1.333333, 1.666666])
s2 = pd.Series([1.333, 1.666])
from pandas.testing import assert_series_equal
assert_series_equal(s1,s2)
raises an AssertionError
. So use the check_less_precise
flag
assert_series_equal(s1,s2, check_less_precise= True) # No assertion error
This doesn't raise an AssertionError
as check_less_precise
only compares 3 digits after decimal.
Not good to use asserts but if you want to avoid numpy, this is a way.
Note: I'm posting this mostly because I came to this thread via a Google search of something similar and it seemed too long for a comment. Not necessarily the best solution nor strictly "ε of precision"-based, but an alternative using scaling and rounding if you want to do this for vectors (i.e. rows) rather than scalars for a DataFrame
(rather than Series
) without looping through explicitly:
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
Xcomb = pd.concat((X, X2), axis=0, ignore_index=True)
# scale
scaler = MinMaxScaler()
scaler.fit(Xcomb)
Xscl = scaler.transform(Xcomb)
# round
df_scl = pd.DataFrame(np.round(Xscl, decimals=8), columns=X.columns)
# post-processing
n_uniq = df_scl.drop_duplicates().shape[0]
n_dup = df.shape[0] + df2.shape[0] - n_uniq
print(f"Number of shared rows: {n_dup}")
np.allclose(s1, s2)
Set the threshold parameters, the docs explain it well.np.isclose()
to return an element-wise boolean series