71

I have a few (almost ten) Gb of memory taken by the ipython kernel. I think this is coming from large objects (matrices, lists, numpy arrays, ...) that I might have produced during some operation and now I do not need anymore.

I would like to list all of the objects I have defined and sort them by their memory footprint. Is there a simple way to do that? For certain types there is nbytes method, but not for all ... so I am looking for a general way to list all objects I have made and their memory occupation.

6
  • 1
    import sys; sorted([(x, sys.getsizeof(x)) for x in dir()], key=lambda x: x[1], reverse=True)?
    – Abdou
    Dec 6, 2016 at 11:01
  • Possible duplicate of How can I check the memory usage of objects in iPython?
    – amin
    Dec 6, 2016 at 11:10
  • 3
    Not a duplicate, he's asking how to find objects he doesn't know the name of. @Abdou you should make that an answer I think.
    – Daniel F
    Dec 6, 2016 at 11:24
  • 1
    Actually should be: import sys; sorted([(x, sys.getsizeof(globals().get(x))) for x in dir()], key=lambda x: x[1], reverse=True). The previous one was getting the sizes of the strings. This one actually gets the objects.
    – Abdou
    Dec 6, 2016 at 13:52
  • 2
    getsizeof for a list is just the size of the pointers in the list, not the size of the objects pointed to. stackoverflow.com/questions/33052836/…
    – hpaulj
    Dec 6, 2016 at 17:12

2 Answers 2

110

Assuming that you are using ipython or jupyter, you will need to do a little bit of work to get a list all of the objects you have defined. That means taking everything available in globals() and filtering out objects that are modules, builtins, ipython objects, etc. Once you are sure you have those objects, then you can proceed to grabbing their sizes with sys.getsizeof. This can be summed up as follows:

import sys

# These are the usual ipython objects, including this one you are creating
ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']

# Get a sorted list of the objects and their sizes
sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)

Please keep in mind that for python objects (those created with python's builtin functions), sys.getsizeof will be very accurate. But it can be a bit inaccurate on objects created using third-party libraries. Furthermore, please be mindful that sys.getsizeof adds an additional garbage collector overhead if the object is managed by the garbage collector. So, some things may look a bit heavier than they actually are.

As a side note, numpy's .nbytes method can be somewhat misleading in that it does not include memory consumed by non-element attributes of the array object.

3
  • 2
    super working code.. tried lot of other places but the size values were incorrect. thanks a ton!!
    – rishi jain
    Jun 11, 2020 at 12:19
  • 1
    Didn't seem to work for me — I dont use ipython so modified like so sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules], key=lambda x: x[1], reverse=True) (no ipython_vars) but the value of hi is 1 and it registered as 28...? Is this the garbage collector you were talking about, because i has a value of 43892978 and it registers as 28 too. Jun 27, 2020 at 14:28
  • fantastic, I had no idea how to list these or what the default objects were, thank you.
    – jimh
    Dec 6, 2022 at 20:40
2

I like the answer @Abdou provided! I would only add the following suggestion. Instead of a list of tuples, I would convert it to a dictionary.

import sys

# These are the usual ipython objects, including this one you are creating
ipython_vars = ["In", "Out", "exit", "quit", "get_ipython", "ipython_vars"]

# Get a sorted list of the objects and their sizes
mem = {
    key: value
    for key, value in sorted(
        [
            (x, sys.getsizeof(globals().get(x)))
            for x in dir()
            if not x.startswith("_") and x not in sys.modules and x not in ipython_vars
        ],
        key=lambda x: x[1],
        reverse=True,
    )
}

Then if I wanted to get the total amount in MBs, all I'd have to do is:

sum(mem.values()) / 1e6
2
  • 1
    Great stuff @daniel-cárdenas. Perhaps, you meant this sum(mem.values())/1e6
    – djoguns
    Jan 31 at 16:39
  • 1
    Correct! Nice catch @djoguns Feb 1 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.