I have the following code, that creates a million objects of a class foo:
for i in range(1000000):
bar = foo()
list_bar.append(bar)
The bar object is only 96 bytes, as determined by getsizeof()
. However, the append step takes almost 8GB of ram. Once the code exits the loop, the ram usage drops to expected amounts (size of the list + some overhead ~103MB). Only while the loop is running does the ram usage skyrocket. Why does this happen? Any workarounds?
PS: Using a generator is not an option, it has to be a list.
EDIT: xrange
doesn't help, using Python 3. The memory usage stays high only during the loop execution, and drops after the loop is through. Could append
have some non-obvious overhead?
xrange
(a generator) instead ofrange
not an option? A list with a million elements will take up a lot of RAM; there's no way around that. – sapi Mar 11 '15 at 11:48list_bar = [foo() for _ in xrange(1000000)]
will reduce the memory overhead, and make your code more idiomatic. – jonrsharpe Mar 11 '15 at 11:50list_bar
has to be a list? Or that the result of thefor
loop condition expression (i.e.,range
) must be a list? – Kevin Mar 11 '15 at 11:51