24

Here's an MWE of some code I'm using. I slowly whittle down an initial dataframe via slicing and some conditions until I have only the rows that I need. Each block of five rows actually represents a different object so that, as I whittle things down, if any one row in each block of five meets the criteria, I want to keep it -- this is what the loop over keep.index accomplishes. No matter what, when I'm done I can see that the final indices I want exist, but I get an error message saying "IndexError: positional indexers are out-of-bounds." What is happening here?

import pandas as pd
import numpy as np

temp = np.random.rand(100,5)

df = pd.DataFrame(temp, columns=['First', 'Second', 'Third', 'Fourth', 'Fifth'])

df_cut = df.iloc[10:]

keep = df_cut.loc[(df_cut['First'] < 0.5) & (df_cut['Second'] <= 0.6)]

new_indices_to_use = []
for item in keep.index:
    remainder = (item % 5)
    add = np.arange(0-remainder,5-remainder,1)
    inds_to_use = item + add
    new_indices_to_use.append(inds_to_use)

new_indices_to_use = [ind for sublist in new_indices_to_use for ind in sublist]
final_indices_to_use = []
for item in new_indices_to_use:
    if item not in final_indices_to_use:
        final_indices_to_use.append(item)

final = df_cut.iloc[final_indices_to_use]
8
  • 1
    Have you tried printing final_indices_to_use and verifying it is what you think it is? On your example I get [10, 11, ..., 98, 99] and len(df_cut) gives 90. May 22, 2017 at 22:34
  • 1
    I'm not sure whether to vote to close as a typo or not, but it looks to me like you should be using .loc instead of .iloc in your last line -- .iloc is for positional access, but you want label-based access.
    – DSM
    May 22, 2017 at 22:40
  • 1
    Hmmmm in my MWE, changing it to .loc does solve the problem. I'll have to check if it works in my actual code -- I could swear I tried that already. Why is .iloc not right here? These are indices, and the description of .iloc says it accepts a list of indices.
    – Arnold
    May 22, 2017 at 22:49
  • Also, if I use an opposite condition and then do final = df_cut.drop(df_cut.index[final_indices_to_use]), I get the same error.
    – Arnold
    May 22, 2017 at 22:55
  • 1
    Because df_cut has a length of 90; indices above 89 will give an IndexError. May 22, 2017 at 22:59

1 Answer 1

53

From Pandas documentation on .iloc (emphasis mine):

Pandas provides a suite of methods in order to get purely integer based indexing. The semantics follow closely python and numpy slicing. These are 0-based indexing.

You're trying to use it by label, which means you need .loc

From your example:

>>>print df_cut.iloc[89]
...
Name: 99, dtype: float64

>>>print df_cut.loc[89]
...
Name: 89, dtype: float64
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.