31

I have data in a csv file with dates stored as strings in a standard UK format - %d/%m/%Y - meaning they look like:

12/01/2012
30/01/2012

The examples above represent 12 January 2012 and 30 January 2012.

When I import this data with pandas version 0.11.0 I applied the following transformation:

import pandas as pd
...
cpts.Date = cpts.Date.apply(pd.to_datetime)

but it converted dates inconsistently. To use my existing example, 12/01/2012 would convert as a datetime object representing 1 December 2012 but 30/01/2012 converts as 30 January 2012, which is what I want.

After looking at this question I tried:

cpts.Date = cpts.Date.apply(pd.to_datetime, format='%d/%m/%Y')

but the results are exactly the same. The source code suggests I'm doing things right so I'm at a loss. Does anyone know what I'm doing wrong?

7
  • 2
    Did you use read_csv? Because then you can do it directly while reading in.
    – joris
    Commented May 21, 2013 at 14:21
  • @joris Yes I did use read_csv. Could you tell what function does the date conversion, and does it handle my formatting issues?
    – cms_mgr
    Commented May 21, 2013 at 14:23
  • But coming back to your original question (because this should also work), what version did you use, because for me it works.
    – joris
    Commented May 21, 2013 at 14:51
  • @joris seems to be the case in 11.0 and dev, posted as issue on github Commented May 21, 2013 at 14:59
  • 1
    Yes, accessing a single column will return a Series. So you could try cpts[['Date']].apply(pd.to_datetime, ...) as a workaround (due to the double [ it will return a dataframe with one column). But note that it should also work on a Series (it is a bug it doesn't), and that the easier way is just to call pd.to_datetime(..) on the column directly as @AndyHayden pointed out in his answer or to do the conversion in read_csv.
    – joris
    Commented May 21, 2013 at 19:24

2 Answers 2

29

You can use the parse_dates option from read_csv to do the conversion directly while reading you data.
The trick here is to use dayfirst=True to indicate your dates start with the day and not with the month. See here for more information: http://pandas.pydata.org/pandas-docs/dev/generated/pandas.io.parsers.read_csv.html

When your dates have to be the index:

>>> import pandas as pd
>>> from StringIO import StringIO
>>> s = StringIO("""date,value
... 12/01/2012,1
... 12/01/2012,2
... 30/01/2012,3""")
>>> 
>>> pd.read_csv(s, index_col=0, parse_dates=True, dayfirst=True)
            value
date             
2012-01-12      1
2012-01-12      2
2012-01-30      3

Or when your dates are just in a certain column:

>>> s = StringIO("""date
... 12/01/2012
... 12/01/2012
... 30/01/2012""")
>>> 
>>> pd.read_csv(s, parse_dates=[0], dayfirst=True)
                 date
0 2012-01-12 00:00:00
1 2012-01-12 00:00:00
2 2012-01-30 00:00:00
2
  • 4
    You can also set a custom parser, this works ok for me: df = pd.read_csv("file.csv", parse_dates=['date_column'], date_parser=lambda d: pd.to_datetime(d, format="%Y/%m/%d", errors="coerce"))
    – ruloweb
    Commented Jun 10, 2018 at 20:26
  • @ruloweb: This seems much safer. Commented Jun 5 at 15:42
19

I think you are calling it correctly, and I posted this as an issue on github.

You can just specify the format to to_datetime directly, for example:

In [1]: s = pd.Series(['12/1/2012', '30/01/2012'])

In [2]: pd.to_datetime(s, format='%d/%m/%Y')
Out[2]:
0   2012-01-12 00:00:00
1   2012-01-30 00:00:00
dtype: datetime64[ns]

Update: As OP correctly points out this doesn't work with NaN, if you are happy with dayfirst=True (which works with NaN too):

s.apply(pd.to_datetime, dayfirst=True)

Worth noting that have to be careful using dayfirst (which is easier than specifying the exact format), since dayfirst isn't strict.

8
  • 1
    thanks this solution is appealing, but it doesn't currently work with missing data, which I have. I suspect the coerce argument for pd.to_datetime in dev would fix that but I can't upgrade my work environment until it's a stable release.
    – cms_mgr
    Commented May 21, 2013 at 19:26
  • @cms_mgr What about: s.apply(lambda t: pd.to_datetime(t, format='%d/%m/%Y')), works with NaN. Commented May 21, 2013 at 19:29
  • 1
    That still fiddles with the dates I'm afraid. Looks like it's a bug - think it might be the first I've ever found!
    – cms_mgr
    Commented May 21, 2013 at 19:37
  • 5
    If only someone had standardised an international date format. Oh, wait.
    – cms_mgr
    Commented May 21, 2013 at 19:39
  • @cms_mgr actually I remember testing that earlier (and it not working). I think that may also be a bug... Commented May 21, 2013 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.