2

The following code fills all my memory:

from sys import getsizeof
import numpy

# from http://stackoverflow.com/a/2117379/272471
def getSize(array):
    return getsizeof(array) + len(array) * getsizeof(array[0])


class test():
    def __init__(self):
        pass
    def t(self):
        temp = numpy.zeros([200,100,100])
        A = numpy.zeros([200], dtype = numpy.float64)
        for i in range(200):
            A[i] = numpy.sum( temp[i].diagonal() ) 
        return A

a = test()
memory_usage("before")
c = [a.t() for i in range(100)]
del a
memory_usage("After")
print("Size of c:", float(getSize(c))/1000.0)

The output is:

('>', 'before', 'memory:', 20588, 'KiB  ')
('>', 'After', 'memory:', 1583456, 'KiB  ')
('Size of c:', 8.92)

Why am I using ~1.5 GB of memory if c is ~ 9 KiB? Is this a memory leak? (Thanks)

The memory_usage function was posted on SO and is reported here for clarity:

def memory_usage(text = ''):
    """Memory usage of the current process in kilobytes."""
    status = None
    result = {'peak': 0, 'rss': 0}
    try:
        # This will only work on systems with a /proc file system
        # (like Linux).
        status = open('/proc/self/status')
        for line in status:
            parts = line.split()
            key = parts[0][2:-1].lower()
            if key in result:
                result[key] = int(parts[1])
    finally:
        if status is not None:
            status.close()
    print('>', text, 'memory:', result['rss'], 'KiB  ')
    return result['rss']
4
  • Replace float(getsizeof(c))/1000.0) with float(getSize(c))/1000.0) and you should get a better value.
    – glglgl
    Jun 13, 2013 at 11:17
  • Thanks glglgl, I actually copied the wrong version of my sample code. Corrected.
    – Pie86
    Jun 13, 2013 at 12:32
  • what happens if you put in a del temp just before the return statement in t()?
    – reptilicus
    Jun 13, 2013 at 15:23
  • Doesn't work. As I wrote, the problem comes from a memory leak in the diagonal function of numpy v. 1.7.0. Fixed in v 1.7.1.
    – Pie86
    Jun 13, 2013 at 15:45

3 Answers 3

6

The implementation of diagonal() failed to decrement a reference counter. This issue had been previously fixed, but the change didn't make it into 1.7.0.

Upgrading to 1.7.1 solves the problem! The release notes contain various useful identifiers, notably issue 2969.

The solution was provided by Sebastian Berg and Charles Harris on the NumPy mailing list.

1
  • 2
    Would be helpful to link to documentation of what was actually fixed.
    – merv
    Dec 8, 2017 at 2:24
1

Python allocs memory from the OS if it needs some.

If it doesn't need it any longer, it may or may not return it again.

But if it doesn't return it, the memory will be reused on subsequent allocations. You should check that; but supposedly the memory consumption won't increase even more.

About your estimations of memory consumption: As azorius already wrote, your temp array consumes 16 MB, while your A array consumes about 200 * 8 = 1600 bytes (+ 40 for internal reasons). If you take 100 of them, you are at 164000 bytes (plus some for the list).

Besides that, I have no explanation for the memory consumption you have.

3
  • That's clear, thanks. Nonetheless if I increase the number of times I call a.t() to 200 my process gets killed by the kernel. Why is this happening? I should have 2*9 = 18 KiB of memory usage, isn't it?
    – Pie86
    Jun 13, 2013 at 11:09
  • @Pie86 check my post, 200 numpy matrixes of that size takes atleast 3 GB space, you do not delete the returned matrixes, only the tiny little class that creates them.
    – jcr
    Jun 13, 2013 at 11:32
  • @azorius that's right, but I keep only the small returned matrix. The problem, as you suggested, is the big temporary matrix temp that is not deleted together with the class.
    – Pie86
    Jun 13, 2013 at 12:43
0

I don't think sys.getsizeof returns what you expect

your numpy vector A is 64 bit (8 bytes) - so it takes up (at least)

8 * 200 * 100 * 100 * 100 / (2.0**30) = 1.5625 GB

so at minimum you should use 1.5 GB on the 100 arrays, the last few hundred mg are all the integers used for indexing the large numpy data and the 100 objects

It seems that sys.getsizeof always returns 80 no matter how large a numpy array is:

sys.getsizeof(np.zeros([200,1000,100])) # return 80
sys.getsizeof(np.zeros([20,100,10])) # return 80

In your code you delete a which is a tiny factory object who's t method return huge numpy arrays, you store these huge arrays in a list called c. try to delete c, then you should regain most of your RAM

5
  • 1
    8 * 200 * 100 * 100 = 16000000 B = 15625 kiB = 15.2587890625 MiB, three orders of magnitude smaller than you anticipated. But that counts for one A only, we have 100 of them. An older system might get into trouble with these sizes...
    – glglgl
    Jun 13, 2013 at 11:22
  • @glglgl yes but 15 MB * 100 = 1.5GB... I was a little to trigger happy with my post as I always post a quick post and then edit out ALL the errors, thanks for pointing this out :)
    – jcr
    Jun 13, 2013 at 11:26
  • @glglgl he thinks he has a memmory leak becuse he is loosing 1.6 GB, but even though he deletes a, his class returns 100 numpy matrixes (that he does not delete) so they are still in memory
    – jcr
    Jun 13, 2013 at 11:30
  • Ok, but temp should be removed (deallocated). I don't want to store it as the name suggests. And A is much smaller, it's just 200 numpy.float64 times 100 iterations! 8 * 200 * 100 = 160000 is the size that I want to use.
    – Pie86
    Jun 13, 2013 at 12:36
  • You and glglgl are absolutely right, my bad :)... if I should try to guess what is going on is is the folloing: python have 3 lairs of buckets inside each other where it allocate memory, and it can only 'give back a chunk of memory' to the os if the outer bucket is completely empty... But other than that I am out of idear, a hack solution would be to generate the temp in the init and then resycle it such that you do not generate 100 of them if python is not able to succesfully dealocate them
    – jcr
    Jun 13, 2013 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.