It will save memory according to memory_usage()
which shows memory use by column, or you could just do info()
to see the total memory use of the entire dataframe.
df = pd.DataFrame({ 'x':[0, 0, 1, 0, 0, 3, 0, 0, 0, 5, 0] })
sdf = df.to_sparse(fill_value=0)
df.memory_usage()
x 88
dtype: int64
sdf.memory_usage()
x 24
dtype: int64
It's kind of interesting, the underlying shape of the data seems to be hidden at the dataframe level:
df.shape
(11, 1)
sdf.shape
(11, 1)
But exposed at the column level:
df['x'].shape
(11,)
sdf['x'].shape
(3,)
I couldn't figure out a way to combine a standard column and sparse column in the same dataframe, but I'm certainly not an expert on this, so I don't know if it is possible or not. But as best I can tell from the documentation the sparse dataframes are just wrappers on "ndarray-like" objects, so I think you need to stick to dataframes that are essentially homogeneous numpy arrays.
I haven't played around with it recently, but I think bcolz is more flexible here (in terms of mixing dtypes), but doesn't give you all the pandas dataframe conveniences obviously.
But as long as you keep your columns separate, it does seem like you can do a lot of things in standard pandastic ways, although not everything:
df = pd.DataFrame({ 'x':[0, 0, 1, 0, 0, 3, 0, 0, 0, 5, 0],
'y':[0, 2, 0, 0, 0, 3, 0, 1, 0, 0, 0] })
sdf = df.to_sparse(fill_value=0)
df + sdf # works
df['x'] + sdf['x'] # works
df['x'] + sdf['y'] # doesn't work, b/c shapes different?
But you can do:
df['x'] + sdf['y'].to_dense()
Not quite as convenient, but not too bad and possibly still saves memory overall in that you have to temporaily expand one column of sdf, but not the whole sdf dataframe.