179

I have a very large data frame in python and I want to drop all rows that have a particular string inside a particular column.

For example, I want to drop all rows which have the string "XYZ" as a substring in the column C of the data frame.

Can this be implemented in an efficient way using .drop() method?

0

7 Answers 7

304

pandas has vectorized string operations, so you can just filter out the rows that contain the string you don't want:

In [91]: df = pd.DataFrame(dict(A=[5,3,5,6], C=["foo","bar","fooXYZbar", "bat"]))

In [92]: df
Out[92]:
   A          C
0  5        foo
1  3        bar
2  5  fooXYZbar
3  6        bat

In [93]: df[~df.C.str.contains("XYZ")]
Out[93]:
   A    C
0  5  foo
1  3  bar
3  6  bat
9
  • 43
    Although what you wrote is correct and more readable, a shorter method would be :df[~df.C.str.contains("XYZ")]
    – EdChum
    Feb 23, 2015 at 17:58
  • 1
    When I do this it works perfectly, however it also does not show any rows in which the value was NaN. Is there a way to get those back so that the resulting data frame contains rows that do not contain the desired string and also NaN?
    – bchards
    Mar 12, 2018 at 16:18
  • 2
    how would you do this if instead of "XYZ" you wanted to see if it contained anything inside of a large list of maybe a 1000 different things to look for.
    – 0004
    Sep 14, 2018 at 3:00
  • 5
    I get an error: TypeError: bad operand type for unary ~: 'float', any ideas regarding this issue?
    – ah bon
    Jul 8, 2020 at 2:34
  • 3
    The problem solved by adding na=False
    – ah bon
    Jul 8, 2020 at 2:37
152

If your string constraint is not just one string you can drop those corresponding rows with:

df = df[~df['your column'].isin(['list of strings'])]

The above will drop all rows containing elements of your list

10
  • 1
    How would you do the inverse of this? I want to check if the column value contains any of the strings. pseudo: for string in list_of_strings, check if column contains it
    – radtek
    Apr 23, 2017 at 5:39
  • 7
    Just remove the "~" df = df[df['your column'].isin(['list of strings'])]
    – Kenan
    Jun 26, 2017 at 18:28
  • 3
    What if we don't know the column? Jul 3, 2018 at 13:22
  • 2
    How would you drop from multiple columns instead of just one?
    – Ali P
    May 23, 2019 at 1:15
  • 2
    This worked for me but I realized that 'list of strings' should be the exact string you want to remove
    – eafloresf
    Aug 5, 2019 at 3:17
55

This will only work if you want to compare exact strings. It will not work in case you want to check if the column string contains any of the strings in the list.

The right way to compare with a list would be :

searchfor = ['john', 'doe']
df = df[~df.col.str.contains('|'.join(searchfor))]
28

Slight modification to the code. Having na=False will skip empty values. Otherwise you can get an error TypeError: bad operand type for unary ~: float

df[~df.C.str.contains("XYZ", na=False)]

Source: TypeError: bad operand type for unary ~: float

16
new_df = df[df.C != 'XYZ']

Reference: https://chrisalbon.com/python/data_wrangling/pandas_dropping_column_and_rows/

10

The below code will give you list of all the rows:-

df[df['C'] != 'XYZ']

To store the values from the above code into a dataframe :-

newdf = df[df['C'] != 'XYZ']
1
  • 1
    This Syntax is way more easier to remember. Thanks Jul 25, 2020 at 17:14
4

if you do not want to delete all NaN, use

df[~df.C.str.contains("XYZ") == True]

Not the answer you're looking for? Browse other questions tagged or ask your own question.