2

Consider a Pandas data frame:

import pandas as pd

df = pd.DataFrame({
    'a': pd.Series([1,1,1,2,3]),
    'b': pd.Series(list('asdfg'))
})

I want to return all of the rows with duplicate values for column a, including the first or last row. I can do this with

df[df['a'].duplicated() | df['a'].duplicated(take_last=True)]

Is there a better way?

2
  • Do you want only first and last row of a or all duplicates for a?
    – Zero
    Jun 12, 2015 at 17:16
  • @JohnGalt all of them, so the first three rows of the example. Jun 12, 2015 at 17:17

1 Answer 1

2

You can count occurrences of a and return values>1 for duplicated rows.

In [25]: df[(df.groupby('a').transform('count')>1).values]
Out[25]:
   a  b
0  1  a
1  1  s
2  1  d
2
  • I ran some benchmarks and it turns out this is 100x slower (on a data frame with ~ 500 rows) than my original solution. Not a big deal in my case but interesting. Jun 12, 2015 at 17:25
  • Yes. duplicated() is a faster and more direct implementation in Cython, read Wes's blog on this.
    – Zero
    Jun 12, 2015 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.