0

This code creates 50k rows containing numpy ndarrays (it takes more than 8 minutes for a 1.5GB file):

import numpy as np, pandas as pd
x = pd.DataFrame(columns=['a', 'b'])
for i in range(100000):
    print(i)
    x.loc['t%i' % i] = [np.random.rand(2000), np.random.rand(2000)]   # not efficient at all
                                                     # the higher i, the longer it takes!
                                                     # like if it concatenates x with a new dataframe each time
x.to_parquet('test.parquet')

As mentioned in Scaling to large datasets, you can load only certain columns:

x = pd.read_parquet("test.parquet", column="a")

but in order to save time, can you load only a specific row, for example x['t123'], without reading the whole file in memory? pd.read_parquet("test.parquet", index="t123") does not exist in the API.

Also, how can we open a 100 GB parquet file, add just one more row, and save it back to disk without rewriting the whole 100 GB file?

(Lastly, x.loc['t1234'] = [np.random.rand(100, 100), np.random.rand(100, 100)] ; x.to_parquet('test.parquet') does not work because parquet cannot serialize numpy 2D or 3D ndarrays, just numpy 1D arrays... This confirms parquet is probably not the right data structure for this data store)

2
  • 1
    The terms you are maybe looking for is "random access", specifically "parquet random access read" and "parquet random access write". Both do not yield many results when googling. Primarily stackoverflow.com/q/66217102/2442804
    – luk2302
    Jun 24, 2022 at 10:47
  • @luk2302 Here is what I want to achieve: afewthingz.com/ndarraydatastore. In case you have an idea, thanks in advance!
    – Basj
    Jun 25, 2022 at 8:53

1 Answer 1

0

If your actual dataframe is numpy arrays you can split your numpy values and save them as npz files.Same methodology goes for dataframes containing strings.You can save and load chucks of it as pickle(DataFrame.to_pickle).Sooner or later if we talking about these numbers a split chuncks will be required most probably. Naming of files ofcourse plays a big role on your algorithm to find the target file range.

2
  • In my case they are mostly always numbers. I'd like to avoid to have to split into many different chunk files manually. I prefer to just have a big file (or ok, many files, but this should be automatically split by the library I'm using). Is there something in Pandas or Parquet or another to deal about this for me automatically?
    – Basj
    Jun 24, 2022 at 10:56
  • there is a chunksize=arg parameter for csv files for writing and reading,but tbh i don't know if there is an easier way for your set up. Jun 24, 2022 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.