Pandas 2.0 update
As part of a broad attempt to make datetime handling in Pandas 2.0 more reliable, date parsing (especially on CSV files) has seen a number of backwards-incompatible changes and deprecations. The infer_datetime_format
and date_parser
parameters reccomended by many other answers are now both deprecated (see PDEP-4 and this issue for the reasons why).
The proper way of parsing dates of known format now is now to use the parse_dates
and date_format
parameters of pd.read_csv()
.
Single column with auto-detected format
df = pd.read_csv(
infile,
parse_dates=['My DateTime']
)
Single column with known format
df = pd.read_csv(
infile,
parse_dates=['My DateTime'],
date_format={'My DateTime': '%Y-%m-%d %H:%M:%S'}
)
Merged columns with known format
df = pd.read_csv(
infile,
parse_dates={'mydatetime': ['My Date', 'My Time']},
# mydatetime will contain my_date and my_time separated by a single space
date_format={'mydatetime': '%Y-%m-%d %H:%M:%S'}
)
If your date parsing logic is more complex than what can be done with static format strings, such as if you have multiple date formats on the same column, or were making advanced use of date_parser
, you are encouraged by the documentation for read_csv
(look at the parse_dates
parameter) to leave your column as the default object
type while reading the file and to do the date conversion in a second pass with pd.to_datetime
, as shown below.
Pandas v1 answer
In addition to what the other replies said, if you have to parse very large files with hundreds of thousands of timestamps, date_parser
can prove to be a huge performance bottleneck, as it's a Python function called once per row. You can get a sizeable performance improvements by instead keeping the dates as text while parsing the CSV file and then converting the entire column into dates in one go:
# For a data column
df = pd.read_csv(infile, parse_dates={'mydatetime': ['date', 'time']})
df['mydatetime'] = pd.to_datetime(df['mydatetime'], exact=True, cache=True, format='%Y-%m-%d %H:%M:%S')
# For a DateTimeIndex
df = pd.read_csv(infile, parse_dates={'mydatetime': ['date', 'time']}, index_col='mydatetime')
df.index = pd.to_datetime(df.index, exact=True, cache=True, format='%Y-%m-%d %H:%M:%S')
# For a MultiIndex
df = pd.read_csv(infile, parse_dates={'mydatetime': ['date', 'time']}, index_col=['mydatetime', 'num'])
idx_mydatetime = df.index.get_level_values(0)
idx_num = df.index.get_level_values(1)
idx_mydatetime = pd.to_datetime(idx_mydatetime, exact=True, cache=True, format='%Y-%m-%d %H:%M:%S')
df.index = pd.MultiIndex.from_arrays([idx_mydatetime, idx_num])
For my use case on a file with 200k rows (one timestamp per row), that cut down processing time from about a minute to less than a second.