This code creates 50k rows containing numpy ndarray
s (it takes more than 8 minutes for a 1.5GB file):
import numpy as np, pandas as pd
x = pd.DataFrame(columns=['a', 'b'])
for i in range(100000):
print(i)
x.loc['t%i' % i] = [np.random.rand(2000), np.random.rand(2000)] # not efficient at all
# the higher i, the longer it takes!
# like if it concatenates x with a new dataframe each time
x.to_parquet('test.parquet')
As mentioned in Scaling to large datasets, you can load only certain columns:
x = pd.read_parquet("test.parquet", column="a")
but in order to save time, can you load only a specific row, for example x['t123']
, without reading the whole file in memory? pd.read_parquet("test.parquet", index="t123")
does not exist in the API.
Also, how can we open a 100 GB parquet file, add just one more row, and save it back to disk without rewriting the whole 100 GB file?
(Lastly, x.loc['t1234'] = [np.random.rand(100, 100), np.random.rand(100, 100)] ; x.to_parquet('test.parquet')
does not work because parquet cannot serialize numpy 2D or 3D ndarrays, just numpy 1D arrays... This confirms parquet is probably not the right data structure for this data store)