1

When using pd.to_datetime on my data frame I get this error:

Out of bounds nanosecond timestamp: 30-04-18 00:00:00

Now from looking on StackO I know I can simply use the coerce option:

pd.to_datetime('13000101', format='%Y%m%d', errors='coerce')

But I was wondering if anyone had an idea on how I might replace these values with a fixed value? Say 1900-01-01 00:00:00 (or maybe 1955-11-12 for anyone who gets the reference!)

Reason being that this data frame is part of a process that handles thousands and thousands of JSONs per day. I want to be able to see in the dataset easily the incorrect ones by filtering for said fixed date.

It is just as invalid for the JSON to contain any date before 2010 so using an earlier date is fine and it is also perfectly acceptable to have a blank (NA) date value so I can't rely on just blanking the data.

1 Answer 1

1

Replace missing values by some default datetime value in Series.mask only for missing values generated by to_datetime with errors='coerce':

df=pd.DataFrame({"date": [np.nan,'20180101','20-20-0']})

t = pd.to_datetime('1900-01-01')
date = pd.to_datetime(df['date'], format='%Y%m%d', errors='coerce')

df['date'] = date.mask(date.isna() & df['date'].notna(), t)
print (df)
        date
0        NaT
1 2018-01-01
2 1900-01-01
2
  • 1
    I did think this but I can't fill all missing values as some could be actually missing from the dataset and I have to have those remain blank.
    – ck3mp
    Feb 16, 2021 at 8:52
  • What's wrong with simply df['date'] = df['date'].fillna(t) ? Oh I see, 'meaningful' NAs already present in 'date' column should be left as NA.
    – smci
    May 6, 2021 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.