4

I am using a pandas dataframe in the sklearn cross_validation train_test_split module.

d=pandas.DataFrame({'a':np.random.randn(300),
                    'c':np.array([el for el in np.ones(100)]+
                                 [el for el in np.zeros(200)])})
from sklearn import cross_validation
(X,y)=(d['a'],d['c'])

This works

X_train_and_cv, X_test,y_train_and_cv,y_test = sklearn.cross_validation.train_test_split(X,y,test_size=0.2,random_state=0)
X_train, X_cv,y_train,y_cv = sklearn.cross_validation.train_test_split(X_train_and_cv,y_train_and_cv,test_size=0.2,random_state=0)

Why doesn't this work?

X_train_and_cv, X_test,y_train_and_cv,y_test = sklearn.cross_validation.train_test_split(X,y,test_size=0.2,random_state=0,stratify=y)
X_train, X_cv,y_train,y_cv = sklearn.cross_validation.train_test_split(X_train_and_cv,y_train_and_cv,test_size=0.2,random_state=0,stratify=y)

in _is_valid_list_like(self, key, axis)
   1536         l = len(ax)
   1537         if len(arr) and (arr.max() >= l or arr.min() < -l):
-> 1538             raise IndexError("positional indexers are out-of-bounds")
   1539 
   1540         return True

IndexError: positional indexers are out-of-bounds

1 Answer 1

3

TL;DR: Your second call to train_test_split uses a different array length for stratify than the y you use. Use stratify=y_train_and_cv.


First, a little side note: cross_validation (0.17.1 docs here) will be deprecated soon, you should use model_selection.train_test_split (0.18.1) instead. I'll import train_test_split itself to shorten the length of what follows:

# Same as this in older versions:
# from sklearn.cross_validation import train_test_split
from sklearn.model_selection import train_test_split 

This is fine:

X_train_and_cv, X_test,y_train_and_cv,y_test = train_test_split(X,y,
                                                                test_size=0.2,
                                                                random_state=0,
                                                                stratify=y)

This is not fine since y=y_train_and_cv(len=240) stratify=y (len=300)

X_train, X_cv,y_train,y_cv = train_test_split(X_train_and_cv,
                                              y_train_and_cv,
                                              test_size=0.2,
                                              random_state=0,
                                              stratify=y)

replace it by:

X_train, X_cv,y_train,y_cv = train_test_split(X_train_and_cv,
                                              y_train_and_cv,
                                              test_size=0.2,
                                              random_state=0,
                                              stratify=y_train_and_cv)
2
  • Wow, I now realize that I was interpreting y as a string instead of variable argument--e.g., stratify = 'yes' --and assuming that it was inferring the to-stratify-on array by the second argument..
    – User0
    Nov 17, 2016 at 14:41
  • Ah! That would have been stratify=True :) Nov 17, 2016 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.