I have a rather large pandas dataframe (1.7G) from which I am selecting some columns to do some computaton (find maximum value of the three selected columns). It seems that this operation is memory intensive. I am trying to find a way to avoid this memory overhead.
For the purpose to this question, I a simplifying the dataframe and using fake data. My code and the memory footprint is shown below,
from memory_profiler import profile
import pandas as pnd
import random
@profile
def main():
cols = [chr(i) for i in range(65,91)]
d = {}
n = 1000000
for c in cols:
d[c] = [random.randint(0,100) for i in range(n)]
df = pnd.DataFrame(d)
items = ['A','F','G']
a = df[items]
b = a.max(axis=0)
if __name__ == "__main__":
main()
Line # Mem usage Increment Line Contents
================================================
6 42.3 MiB 0.0 MiB @profile
7 def main():
8 42.3 MiB 0.0 MiB cols = [chr(i) for i in range(65,91)]
9 42.3 MiB 0.0 MiB d = {}
10 42.3 MiB 0.0 MiB n = 1000000
11 240.6 MiB 198.3 MiB for c in cols:
12 240.6 MiB 0.0 MiB d[c] = [random.randint(0,100) for i in range(n)]
13 446.7 MiB 206.1 MiB df = pnd.DataFrame(d)
14 446.7 MiB 0.0 MiB items = ['A','F','G']
15 469.7 MiB 23.1 MiB a = df[items]
16 469.8 MiB 0.1 MiB b = a.max(axis=0)
In the above operation, it seems that df[items] uses up 23MB of memory. I am speculating that this because it is making a copy of the df and placing it in 'a'.
Is there a way get rid of this memory overhead when selecting columns?