2
x_train:(153347,53)
x_test:(29039,52)
y:(153347,)

I am working with sklearn. To cross validate and reshape my dataset i did:

x_train, x_test, y_train, y_test = cross_validation.train_test_split(
x, y, test_size=0.3)

x_train = np.pad(x, [(0,0)], mode='constant')
x_test = np.pad(x, [(0,0)], mode='constant')
y = np.pad(y, [(0,0)], mode='constant')
x_train = np.arange(8127391).reshape((-1,1))
c = x.T
np.all(x_train == c)
x_test = np.arange(1510028).reshape((-1,1))
c2 = x.T
np.all(x_test == c2)
y = np.arange(153347).reshape((-1,1))
c3 = x.T
np.all(y == c3)

My error message is:ValueError: Found arrays with inconsistent numbers of samples: [ 2 153347]

I am not sure i need to pad my dataset in this case and the reshape is not working. Any ideas on how i can fix this?

3
  • Perhaps you should mention that you use sklearn and describe more what you're doing. Otherwise, chances are high that it is simply ignored.
    – fricke
    Oct 1, 2016 at 9:18
  • Ok, thanks for the tip.
    – Bolajio
    Oct 1, 2016 at 9:30
  • where is this message originating from? It seems that you simply passed a transposed y
    – lejlot
    Oct 1, 2016 at 9:33

1 Answer 1

1

With the little we see here one, I believe the call to cross_validation.train_test_split dumps because the the length of the two vectors does not coincide. So for every X (the data tuple we you observe) you need a Y (the data-point that is observed as a result).

At least this leads to the error shown above.

You should definitely improve on the formulation of the problem. Very much so.

regards, fricke

4
  • I see what you are saying, the problem is calling y_test since it does not exist in the dataframe. Thanks!
    – Bolajio
    Oct 1, 2016 at 10:16
  • if this solves the problem could you accept the solution, thanks
    – fricke
    Oct 1, 2016 at 10:35
  • Ok one last thing if you can help. Do i need to merge separate files of train and test to perform a cross validation? I have the csv as a train csv and a test csv. Can i just merge them into one and program based on separating my dataset as x and y instead?
    – Bolajio
    Oct 1, 2016 at 11:44
  • Yes, at least this is a way to achieve a cross validation. But Sklearn offers you a very convenient method of doing it by passing the entire learning and test-set to the function 'cross_val_score'. This function then splits entire set into a series of disjoint learning and test sets and does an average on the score.
    – fricke
    Oct 1, 2016 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.