This is quite a common question for beginners when making multiple conditions in Pandas. Generally speaking, there are two possible conditions causing this error:
Condition 1: Python Operator Precedence
There is a paragraph of Boolean indexing | Indexing and selecting data — pandas documentation explains this
Another common operation is the use of boolean vectors to filter the data. The operators are: |
for or
, &
for and
, and ~
for not
. These must be grouped by using parentheses.
By default Python will evaluate an expression such as df['A'] > 2 & df['B'] < 3
as df['A'] > (2 & df['B']) < 3
, while the desired evaluation order is (df['A'] > 2) & (df['B'] < 3)
.
# Wrong
df['col'] < -0.25 | df['col'] > 0.25
# Right
(df['col'] < -0.25) | (df['col'] > 0.25)
There are some possible ways to get rid off the parentheses, I will cover this later.
Condition 2: Improper Operator/Statement
As is explained in previous quotation, you need use |
for or
, &
for and
, and ~
for not
# Wrong
(df['col'] < -0.25) or (df['col'] > 0.25)
# Right
(df['col'] < -0.25) | (df['col'] > 0.25)
Another possible situation is that you are using a boolean Series in if
statement.
# Wrong
if pd.Series([True, False]):
pass
It's clear that Python if
statement accepts boolean like expression rather than Pandas Series. You should use pandas.Series.any
or methods listed in the error message to convert the Series to a value according to your need.
For example:
# Right
if df['col'].eq(0).all():
# If you want all column values equal to zero
print('do something')
# Right
if df['col'].eq(0).any():
# If you want at least one column value equal to zero
print('do something')
Let's talk about ways to escape the parentheses in the first situation.
- Use Pandas mathematical functions
Pandas has defined a lot of mathematical functions including comparison as follows:
As a result, you can use
df = df[(df['col'] < -0.25) | (df['col'] > 0.25)]
# is equal to
df = df[df['col'].lt(-0.25) | df['col'].gt(0.25)]
- Use
pandas.Series.between()
If you want to select rows in between two values, you can use pandas.Series.between
df['col].between(left, right)
is equal to
(left <= df['col']) & (df['col'] <= right)
;
df['col].between(left, right, inclusive='left)
is equal to
(left <= df['col']) & (df['col'] < right)
;
df['col].between(left, right, inclusive='right')
is equal to
(left < df['col']) & (df['col'] <= right)
;
df['col].between(left, right, inclusive='neither')
is equal to
(left < df['col']) & (df['col'] < right)
;
df = df[(df['col'] > -0.25) & (df['col'] < 0.25)]
# is equal to
df = df[df['col'].between(-0.25, 0.25, inclusive='neither')]
- Use
pandas.DataFrame.query()
Document referenced before has a chapter The query()
Method explains this well.
pandas.DataFrame.query()
can help you select a DataFrame with a condition string. Within the query string, you can use both bitwise operators(&
and |
) and their boolean cousins(and
and or
). Moreover, you can omit the parentheses, but I don't recommend for readable reason.
df = df[(df['col'] < -0.25) | (df['col'] > 0.25)]
# is equal to
df = df.query('col < -0.25 or col > 0.25')
- Use
pandas.DataFrame.eval()
pandas.DataFrame.eval()
evaluates a string describing operations on DataFrame columns. Thus, we can use this method to build our multiple condition. The syntax is same with pandas.DataFrame.query()
.
df = df[(df['col'] < -0.25) | (df['col'] > 0.25)]
# is equal to
df = df[df.eval('col < -0.25 or col > 0.25')]
pandas.DataFrame.query()
and pandas.DataFrame.eval()
can do more things than I describe here, you are recommended to read their documentation and have fun with them.
|
instead ofor
abs(result['var'])>0.25
max()
function. Replacing it with withnumpy.maximum()
for element-wise maxima between two values solved my problem.