14

I am using the multiprocessing functions of Python to run my code parallel on a machine with roughly 500GB of RAM. To share some arrays between the different workers I am creating a Array object:

N = 150
ndata = 10000
sigma = 3
ddim = 3

shared_data_base = multiprocessing.Array(ctypes.c_double, ndata*N*N*ddim*sigma*sigma)
shared_data = np.ctypeslib.as_array(shared_data_base.get_obj())
shared_data = shared_data.reshape(-1, N, N, ddim*sigma*sigma)

This is working perfectly for sigma=1, but for sigma=3 one of the harddrives of the device is slowly filled, until there is no free space anymore and then the process fails with this exception:

OSError: [Errno 28] No space left on device

Now I've got 2 questions:

  1. Why does this code even write anything to the disc? Why isn't it all stored in the memory?
  2. How can I solve this problem? Can I make Python store it entireley in the RAM without writing it to the HDD? Or can I change the HDD on which this array is written?

EDIT: I found something online which suggests, that the array is stored in the "shared memory". But the /dev/shm device has plenty more free space as the /dev/sda1 which is filled up by the code above. Here is the (relevant part of the) strace log of this code.

Edit #2: I think that I have found a workarround for this problem. By looking at the source I found that multiprocessing tries to create a temporary file in a directory which is determinded by using

process.current_process()._config.get('tempdir')

Setting this value manually at the beginning of the script

from multiprocessing import process
process.current_process()._config['tempdir'] =  '/data/tmp/'

seems to be solving this issue. But I think that this is not the best way to solve it. So: are there any other suggestions how to handle it?

14
  • 1
    Run it under strace to see what is going on.
    – user933161
    Apr 23, 2017 at 16:16
  • 3
    BTW, where can one get 500G of RAM? :)
    – user933161
    Apr 23, 2017 at 16:17
  • Also error code might be wrong. And really meaning out of memory. I. e. python library abuses that code.
    – user933161
    Apr 23, 2017 at 16:21
  • My guess is that multiprocessing.Array() uses /dev/shm, which (at least on Linux) is limited to half the available RAM (check with df -kh /dev/shm). Look here on how to increase it (if that's the limiting factor).
    – robertklep
    Apr 23, 2017 at 16:23
  • 2
    Are you sure sizeof(c_double) * ndata*N*N*ddim*sigma*sigma fits into your RAM?
    – user933161
    Apr 23, 2017 at 16:25

1 Answer 1

2

These data are larger than 500GB. Just shared_data_base would be 826.2GB on my machine by sys.getsizeof() and 1506.6GB by pympler.asizeof.asizeof(). Even if they were only 500GB, your machine needs some of that memory in order to run. This is why the data are going to disk.

import ctypes
from pympler.asizeof import asizeof
import sys


N = 150
ndata = 10000
sigma = 3
ddim = 3
print(sys.getsizeof(ctypes.c_double(1.0)) * ndata*N*N*ddim*sigma*sigma)
print(asizeof(ctypes.c_double(1.0)) * ndata*N*N*ddim*sigma*sigma)

Note that on my machine (Debian 9), /tmp is the location that fills. If you find that you must use disk, be certain that the location on disk used has enough available space, typically /tmp isn't a large partition.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.