29
import numpy as np

array = np.zeros((210000, 210000)) # default numpy.float64
array.nbytes

When I run the above code on my 8GB memory MacBook with macOS, no error occurs. But running the same code on a 16GB memory PC with Windows 10, or a 12GB memory Ubuntu laptop, or even on a 128GB memory Linux supercomputer, the Python interpreter will raise a MemoryError. All the test environments have 64-bit Python 3.6 or 3.7 installed.

0

1 Answer 1

26

@Martijn Pieters' answer is on the right track, but not quite right: this has nothing to do with memory compression, but instead it has to do with virtual memory.

For example, try running the following code on your machine:

arrays = [np.zeros((21000, 21000)) for _ in range(0, 10000)]

This code allocates 32TiB of memory, but you won't get an error (at least I didn't, on Linux). If I check htop, I see the following:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
31362 user       20   0 32.1T 69216 12712 S  0.0  0.4  0:00.22 python

This because the OS is perfectly willing to overcommit on virtual memory. It won't actually assign pages to physical memory until it needs to. The way it works is:

  • calloc asks the OS for some memory to use
  • the OS looks in the process's page tables, and finds a chunk of memory that it's willing to assign. This is fast operation, the OS just stores the memory address range in an internal data structure.
  • the program writes to one of the addresses.
  • the OS receives a page fault, at which point it looks and actually assigns the page to physical memory. A page is usually a few KiB in size.
  • the OS passes control back to the program, which proceeds without noticing the interruption.

Creating a single huge array doesn't work on Linux because, by default, a "heuristic algorithm is applied to figure out if enough memory is available". (thanks @Martijn Pieters!) Some experiments on my system show that for me, the kernel is unwilling to provide more than 0x3BAFFFFFF bytes. However, if I run echo 1 | sudo tee /proc/sys/vm/overcommit_memory, and then try the program in the OP again, it works fine.

For fun, try running arrays = [np.ones((21000, 21000)) for _ in range(0, 10000)]. You'll definitely get an out of memory error, even on MacOs or Linux with swap compression. Yes, certain OSes can compress RAM, but they can't compress it to the level that you wouldn't run out of memory.

9
  • I tried your first example which indeed the Linux allocated 32t virtual memory on a 128GB memory server. However, MemoryError raised with my example array = np.zeros((210000, 210000)). My example will only need 352GB virtual memory which seems more reasonable than the 32t virtual memory. Mar 3, 2019 at 1:37
  • @BlaiseWang Right, I addressed that in my answer "I have no idea why creating a single huge array doesn't work on Linux or Windows, but I'd expect it to have more to do with the platform's implementation of libc and the limits imposed there than the operating system." If you'd really like to know why, I'd suggest you review the code in code.woboq.org/userspace/glibc/malloc/malloc.c.html (I can't be bothered to do so)
    – flaviut
    Mar 3, 2019 at 1:58
  • @BlaiseWang It's because NumPy is filling the array with zeroes. malloc() typically doesn't care about what's in the allocated memory when it gets it, so Unix might do the same with the pages it returns - as soon as you write zeroes, though, it gets allocated. Some versions of Unix, like the one on user60651's machine, might also guarantee that new pages are zeroed out, and not allocate memory unless the written value is not a zero (and just return zero if an unwritten page is read).
    – TheHans255
    Mar 3, 2019 at 3:18
  • @TheHansinator It's a reasonable guess but still this cannot explain why 32TiB virtual memory can be allocated but 350GB cannot be. Mar 3, 2019 at 3:53
  • 1
    Indeed, my interpretation of the compression system was wrong, it applies to inactive pages in physical memory, not to VM unaddressed pages of the current process. The VM overcommitting behaviour of the MacOS kernel is very hard to track down (Apple stopped updating their Virtual Memory and kernel internals documentation in 2013), but since the behaviour is configurable in Linux perhaps the OP should experiment with adjusting the overcommit policy.
    – Martijn Pieters
    Mar 3, 2019 at 9:30

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.