I have a very large Pandas dataframe that I would like to save to disk to use later. The dataframe only contains string data.
However, no matter what format I use, the saving process crashes my Google Colab enviroment due to using up all available RAM, except CSV, which doesn't complete even after 5 hours.
but that also crashes the enviroment.
Is there a workaround to saving a large text pandas dataframe to disk?
I have tried to_json
, to_feather
, to_parquet
, to_pickle
, and they all crash the enviroment.
I also tried to_sql
by using
from sqlalchemy import create_engine
engine = sqlalchemy.create_engine("sqlite:///database.db")
df.to_sql("table", engine)
I would like to save my dataframe to disk within a reasonable time without crashing the enviroment.