9

I have the following snippet of Python code:

import pandas as pd

# print normal index
print data.index

# convert from df to JSON and back
data_json = data.to_json()
df = pd.read_json(data_json)
df.index = pd.to_datetime(df.index)
print df.index

for some reason running this returns in:

<class 'pandas.tseries.index.DatetimeIndex'>
[1950-01-03 00:00:00, ..., 2014-08-21 00:00:00]
Length: 16264, Freq: None, Timezone: None
<class 'pandas.tseries.index.DatetimeIndex'>
[1966-10-31 00:00:00, ..., 2001-09-07 00:00:00]
Length: 16264, Freq: None, Timezone: None

Can someone explain to me what is going on and how I can have the index persist through the transformations?

3
  • 2
    Can you post some example data where this fails? On a dummy set it worked for me. You should also note versions.
    – chrisb
    Aug 22, 2014 at 20:36
  • print sys.version print pd.__version__ 2.7.5 |Anaconda 1.8.0 (64-bit)| (default, Nov 4 2013, 15:30:26) [GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] 0.12.0 The data is: data = Quandl.get("YAHOO/INDEX_GSPC", trim_start="1950-01-03", trim_end="2014-08-21")
    – L1meta
    Aug 22, 2014 at 20:41
  • 4
    fixed it by upgrading pandas to 0.14.1 and specifying date_format='iso' in the to_json() call.
    – L1meta
    Aug 22, 2014 at 21:08

1 Answer 1

15

The error here is that to_json saves dates with ms resolution by defaul, while to_datetime converts with nanosecond resolution by default. To fix, either of these (but not both!) would work.

pd.to_datetime(df.index, unit='ms')
#OR
data_json = data.to_json(date_unit='ns')

As noted in comments, you can also just save the json with the dates in iso format.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.