1

While hunting for a memory hog in my python code I came across some strange behaviour of numpy.linalg.lstsq. It seems to allocate new memory each time it is called with arrays of a certain size.

The example below uses these functions to get memory usage on Linux.

import numpy as np

def testit(n):
    A = np.random.randn(n, 100)
    b = np.random.randn(n, 10)
    deltamem = []
    for i in range(10):
        before = memory()
        x = np.linalg.lstsq(A, b)
        after = memory()
        deltamem.append(after-before)
    return deltamem

print(testit(100))
print(testit(1024))
print(testit(1025))
print(testit(10000))
print(testit(65630))
print(testit(65640))
print(testit(70000))

Output:

100 : [208896.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
1024 : [302313472.0, 819200.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
1025 : [819200.0, 0.0, 155648.0, 0.0, 159744.0, 0.0, 0.0, 167936.0, 0.0, 159744.0]
10000 : [8564736.0, 16125952.0, 8122368.0, 8122368.0, 8126464.0, 8122368.0, 8126464.0, 8122368.0, 8126464.0, 8122368.0]
65630 : [14950400.0, 68157440.0, 3145728.0, 3145728.0, 3145728.0, 3145728.0, 3145728.0, 3145728.0, 3149824.0, 3145728.0]
65640 : [3153920.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
70000 : [1048576.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

I guess it makes sense that memory is allocated in the first call(s) with a new size. It also seems reasonable that no memory is allocated above a certain size. Presumably there is some internal caching for smaller matrix systems.

What baffles me is the range between 1025 and ~65635:

  • Why does it repeatedly allocate memory?
  • The range where this happens is rather suspicious (>1024 and <65536?)
  • How can I avoid this issue? (The problem I'm solving falls exactly in the troublesome range)

I can reproduce this on my Arch Linux machine with python 3.3.3 and python 2.7.6; numpy version is 1.8.0 in both cases.

Update: This does not occur on a different machine that runs the same configuration of linux/python/numpy. The main difference between the machines is that mine is an older AMD Phenom CPU, while the other is intel.

I no longer believe this is a python/numpy problem - but what is the problem? CPU related optimization in the C lib?

6
  • there are a few working array allocated every time you call lstsq, here is the source code.github.com/numpy/numpy/blob/v1.8.0/numpy/linalg/linalg.py#L1816
    – HYRY
    Jan 20, 2014 at 12:37
  • Why don't they get freed when the function returns? (I also tried to force garbage collection inside the loop without success)
    – MB-F
    Jan 20, 2014 at 13:07
  • How does your memory() measure the memory usage? It's up to the C library to decide when to decrease the heap size and return freed memory to the system.
    – pv.
    Jan 20, 2014 at 13:28
  • @pv. It parses /proc/<pid>/status to get VmSize, which I believe is the process' virtual memory size. If memory was freed and then allocated again I would expect the C library to reuse that memory if it wasn't returned to the system yet. Unfortunately, the function blows my memory if I repeat the loop several 100 times.
    – MB-F
    Jan 20, 2014 at 14:17
  • 1
    I do not see such behavior (on Numpy 1.8.0 or 1.7.1). If you have an older version, bugs are in principle possible.
    – pv.
    Jan 21, 2014 at 9:35

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.