25

I found the documentation for pandas.DataFrame.pop, but after trying it and examining the source code, it does not seem to do what I want.

If I make a dataframe like this:

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(10,6))
# Make a few areas have NaN values
df.iloc[1:3,1] = np.nan
df.iloc[5,3] = np.nan
df.iloc[7:9,5] = np.nan


>>> df
          0         1         2         3         4         5
0  0.772762 -0.442657  1.245988  1.102018 -0.740836  1.685598
1 -0.387922       NaN -1.215723 -0.106875  0.499110  0.338759
2  0.567631       NaN -0.353032 -0.099011 -0.698925 -1.348966
3  1.320849  1.084405 -1.296177  0.681111 -1.941855 -0.950346
4 -0.026818 -1.933629 -0.693964  1.116673  0.392217  1.280808
5 -1.249192 -0.035932 -1.330916       NaN -0.135720 -0.506016
6  0.406344  1.416579  0.122019  0.648851 -0.305359 -1.253580
7 -0.092440 -0.243593  0.468463 -1.689485  0.667804       NaN
8 -0.110819 -0.627777 -0.302116  0.630068  2.567923       NaN
9  1.884069 -0.393420 -0.950275  0.151182 -1.122764  0.502117

If I want to remove selected rows and assign them to a separate object in one step, I would want a pop behavior, like this:

# rows in column 5 which have NaN values
>>> df[df[5].isnull()].index
Int64Index([7, 8], dtype='int64')

# remove them from the dataframe, assign them to a separate object
>>> nan_rows = df.pop(df[df[5].isnull()].index)

However, this does not appear to be supported. Instead, it seems like I am forced to do this in two separate steps, which seems a bit inelegant.

# get the NaN rows
>>> nan_rows = df[df[5].isnull()]

>>> nan_rows
          0         1         2         3         4   5
7 -0.092440 -0.243593  0.468463 -1.689485  0.667804 NaN
8 -0.110819 -0.627777 -0.302116  0.630068  2.567923 NaN

# remove from orignal df
>>> df = df.drop(nan_rows.index)

>>> df
          0         1         2         3         4         5
0  0.772762 -0.442657  1.245988  1.102018 -0.740836  1.685598
1 -0.387922       NaN -1.215723 -0.106875  0.499110  0.338759
2  0.567631       NaN -0.353032 -0.099011 -0.698925 -1.348966
3  1.320849  1.084405 -1.296177  0.681111 -1.941855 -0.950346
4 -0.026818 -1.933629 -0.693964  1.116673  0.392217  1.280808
5 -1.249192 -0.035932 -1.330916       NaN -0.135720 -0.506016
6  0.406344  1.416579  0.122019  0.648851 -0.305359 -1.253580
9  1.884069 -0.393420 -0.950275  0.151182 -1.122764  0.502117

Is there a one-step method built-in? Or is this the way you're 'supposed' to do it?

2

2 Answers 2

4

Since you can pop columns, you can take transpose of the dataframe and pop its columns, ie. the rows of the original df like this. Here is the original df.

    import numpy as np
df = pd.DataFrame(np.random.randint(0, 10, size=(3, 3)), columns = ['a', 'b', 'c'])

print(df)
   a  b  c
0  4  9  4
1  5  5  8
2  5  7  4

Then you take transpose of it and pop column 0 which is the row 0 of the original df.

df_t = df.T
popped_row = df_t.pop(0)

Now you have the popped row

print(popped_row)
a    4
b    9
c    4
Name: 0, dtype: int32

And then you have the original dataframe without the first row.

df = df_t.T

print(df)
   a  b  c
1  5  5  8
2  5  7  4
2
  • 2
    but how is this approach any better than the approach OP has? It has more lines of code, is less efficient, and is harder to read Commented Jan 27, 2023 at 0:54
  • I think this is a question of a taste. The original post was after a solution that works like pop method - and I provided such a solution. It is up to you to decide which you like more. Actually both solutions can be expressed on two lines, if that is what you want. I think often it is better not to minimise lines of code
    – MattiH
    Commented Feb 1, 2023 at 16:41
2

pop source code:

    def pop(self, item):
        """
        Return item and drop from frame. Raise KeyError if not found.
        """
        result = self[item]
        del self[item]
        try:
            result._reset_cacher()
        except AttributeError:
            pass

        return result
File:      c:\python\lib\site-packages\pandas\core\generic.py

del definitely won't work if item is not a simple column name. Pass a simple column name, or do it in two steps.

5
  • 1
    yes that is what I meant by what I posted in the OP. pop only works on columns here, not rows. Commented Feb 16, 2017 at 22:56
  • So what do you expect? It s SO not GH :-)
    – Zeugma
    Commented Feb 16, 2017 at 23:07
  • not sure what GH is. It seemed like there would be a one-step method for this which I hadn't found yet. Commented Feb 16, 2017 at 23:29
  • GitHub : you re searching for a feature that is not implemented in pop.
    – Zeugma
    Commented Feb 16, 2017 at 23:31
  • 1
    yes I am searching for other/better implementations of the feature I described Commented Feb 16, 2017 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.