I've been working to_csv()
/read_csv()
to read/write a data frame a user works with in an applet, where one of the columns is a datetime.datetime
object, and it seems like to_csv
automatically converts the datetimes to strings. Is this correct? If so, is there a way to "preserve" the dates as datetime
rather than them being converted to strings? I've read through the documentation, and I can't seem to find the answer. Thank you.
1 Answer
To preserve the exact structure of a DataFrame, complete with data types, check out the pickle module, which "serializes" any python object to disk and reloads it back into a python environment.
Use pd.to_pickle
instead of pd.to_csv
, optionally with a compression
argument (see docs):
# Save to pickle
df.to_pickle('pickle-file.pkl')
# Pickle with compression
df.to_pickle('pickle-file.pkl.gz', compression='gzip')
# Load pickle from disk
df = pd.read_pickle('pickle-file.pkl')
# or...
df = pd.read_pickle('pickle-file.pkl.gz', compression='gzip')
-
-
1It compresses your data, allowing it to take up less space on disk :) more details in the docs: pandas.pydata.org/pandas-docs/stable/generated/… Apr 10, 2018 at 4:52
parse_dates
argument inread_csv
should allow you to read dates back into python objects. Or you can do as the answer suggests if you are only using the csv for serialization.