8

Let's say I have 2 dictionaries in Python, like this:

d1 = {}
d2 = {}
d1[(i, j)] = 10
d2[(i, j)] = 20

Instead I could do it like this:

d = {}
d[(i, j)] = (10, 20)

Accessing it could be done with d[(i,j)][0] and d[(i,j)][1].

What I would like to ask is this:

Does the second option need less memory than the first?

If yes, is it half the memory?

I need to use very large dictionaries in a program I am writing and if the second solution is better I would opt for that.

3
  • Is it half of the memory? - Yes. May 28, 2019 at 15:29
  • I suggest doing some further reading on dictionaries in the python tutorial (subchapter 5.5). Then you can specify better what you want to do. It is quite a different thing if you have two separate dictionaries or only one.
    – lidrariel
    May 28, 2019 at 15:30
  • You should also look at more specialized datastructures like numpy arrays, which are optimized for matrix style data.
    – schlenk
    May 28, 2019 at 15:57

1 Answer 1

7

Tested this on Windows 10 machine in 32bit Python 3.7.3 repl:

This took 155MB of memory:

>>> d1 = {(i, j): 10 for i in range(1000) for j in range(1000)}
>>> d2 = {(i, j): 20 for i in range(1000) for j in range(1000)}

And this only 79MB:

>>> d = {(i, j): (10, 20) for i in range(1000) for j in range(1000)}

So the second one is obviously better in terms of memory usage, but they are quite different solutions and it's hard to say which one is 'better' overall. It all depends on the use case.

 

Edit:

The difference is a lot smaller, but still significant when using random values (with unique ids).

This took 157MB:

>>> from random import randint
>>> d1 = {(i, j): randint(0, 100) for i in range(1000) for j in range(1000)}
>>> d2 = {(i, j): randint(0, 100) for i in range(1000) for j in range(1000)}

And this 119MB:

>>> from random import randint
>>> d = {(i, j): (randint(0, 100), randint(0, 100)) for i in range(1000) for j in range(1000)}
15
  • 1
    This is surprising. Doesn't a tuple with 2 elements take up more memory than the 2 elements themselves?
    – Barmar
    May 28, 2019 at 15:32
  • @Barmar I assume so, but the additional dictionary obviously has a lot of overhead.
    – ruohola
    May 28, 2019 at 15:36
  • 1
    Also surprising, I'd expect that to be asymptotically negligible. Maybe the hash table has lots of extra space to avoid collisions.
    – Barmar
    May 28, 2019 at 15:37
  • Is the second one 1000*1000 tuples, or one tuple and 1000*1000 references to the same tuple? May 28, 2019 at 15:41
  • 1
    @pikpapo: You can check the memory usage yourself. If you don't know how, ask a separate question on StackOverflow.
    – pts
    May 28, 2019 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.