16

Getting this memory error. But the book/link I am following doesn't get this error.

A part of Code:

from sklearn.linear_model import SGDClassifier
sgd_clf = SGDClassifier()
sgd_clf.fit(x_train, y_train)

Error: MemoryError: Unable to allocate 359. MiB for an array with shape (60000, 784) and data type float64

I also get this error when I try to scale the data using StandardScaler's fit_transfrom

But works fine in both if I decrease the size of training set (something like : x_train[:1000] ,y_train[:1000])

Link for the code in the book here. The error I get is in Line 60 and 63 (In [60] and In [63])

The book : Aurélien Géron - Hands-On Machine Learning with Scikit-Learn Keras and Tensorflow 2nd Ed (Page : 149 / 1130)

So here's my question :

Does this has anything to do with my ram? and what does "Unable to allocate 359" mean? is it the memory size ?

Just in case my specs : CPU - ryzen 2400g , ram - 8gb (3.1gb is free when using jupyter notebook)

4
  • 1
    Any update on your problem? Was a reboot helpfull or did you manage to solve the problem otherwise?
    – sns
    Commented Jul 14, 2020 at 4:27
  • 1
    no unfortunately. tried all kinds of stuff still didn't work. Now decided to work with small parts of the dataset until I upgrade my ram Commented Jul 14, 2020 at 10:59
  • 1
    Are you using partial_fit() or just a subset of the dataset?
    – sns
    Commented Jul 14, 2020 at 11:53
  • Used a subset of the dataset. But will try partial_fit() and see how it goes later Commented Jul 14, 2020 at 12:36

4 Answers 4

8

The message is straight forward, yes, it has to do with the available memory.

359 MiB = 359 * 2^20 bytes = 60000 * 784 * 8 bytes

where MiB = Mebibyte = 2^20 bytes, 60000 x 784 are the dimensions of your array and 8 bytes is the size of float64.

Maybe the 3.1gb free memory is very fragmented and it is not possible to allocate 359 MiB in one piece?

A reboot may be helpful in that case.

2
  • 1
    Wow, I've never seen anyone use the IEC/ISO standards; never even heard of Mebibyte before! Commented Feb 14, 2021 at 21:21
  • 1
    Well, sklearn uses it in their memory error messages. That's why I explained it. I have the feeling that these units are getting used more and more as time goes by. Maybe the education material in schools and universities has slowly been updated to make use of the IEC standard and the new generations learn about it.
    – sns
    Commented Feb 15, 2021 at 9:56
2

Upgrading python-64 bit seems to have solved all the "Memory Error" problem.

3
  • 15
    I am using 64 bit and the problem occurs. This answer is not correct
    – apYan
    Commented Mar 19, 2022 at 9:36
  • @apYan here's why 64-bit should fix the problem stackoverflow.com/a/3117794/12485480 double check your python version (from where you're executing the code) before executing the program since you can have multiple python environment in the same machine Commented Sep 8, 2022 at 11:03
  • Im also using 64bit and im getting hte same issue Commented Mar 6 at 0:57
1

did you try converting to smaller sized data types? float64 to float32 or if possible np.uint8 ?

Pred['train'] = Pred['train'].astype(np.uint8,errors='ignore')

1
  • 1
    problem was fixed by installing 64-bit python. 32-bit won't let me allocate memory above a certain range. Commented Jul 31, 2021 at 9:08
-2

Just Restart Your PC After Closing All Tabs And Softwares.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.