15

I am trying to interpolate time series data, df, which looks like:

         id      data        lat      notes    analysis_date
0  17358709       NaN  26.125979      None     2019-09-20 12:00:00+00:00
1  17358709       NaN  26.125979      None     2019-09-20 12:00:00+00:00
2  17352742 -2.331365  26.125979      None     2019-09-20 12:00:00+00:00
3  17358709 -4.424366  26.125979      None     2019-09-20 12:00:00+00:00

I try: df.groupby(['lat', 'lon']).apply(lambda group: group.interpolate(method='linear')), and it throws {ValueError}Invalid fill method. Expecting pad (ffill) or backfill (bfill). Got linear I suspect the issue is with the fact that I have None values, and I do not want to interpolate those. What is the solution?

df.dtypes gives me:

id                                                                int64
data                                                            float64
lat                                                             float64
notes                                                            object
analysis_date         datetime64[ns, psycopg2.tz.FixedOffsetTimezone...
dtype: object
1
  • Just a note: the same error happens if a column is i.e. "Int64". Before interpolation we must coerce it to float: df["x"] = df["x"].astype("float64")
    – Alex Poca
    Jan 16 at 10:50

3 Answers 3

20

DataFrame.interpolate has issues with timezone-aware datetime64ns columns, which leads to that rather cryptic error message. E.g.

import pandas as pd

df = pd.DataFrame({'time': pd.to_datetime(['2010', '2011', 'foo', '2012', '2013'], 
                                          errors='coerce')})
df['time'] = df.time.dt.tz_localize('UTC').dt.tz_convert('Asia/Kolkata')
df.interpolate()

ValueError: Invalid fill method. Expecting pad (ffill) or backfill (bfill). Got linear


In this case interpolating that column is unnecessary so only interpolate the column you need. We still want DataFrame.interpolate so select with [[ ]] (Series.interpolate leads to some odd reshaping)

df['data'] = df.groupby(['lat', 'lon']).apply(lambda x: x[['data']].interpolate())
1

This error happens because one of the columns you are interpolating is of object data type. Interpolating works only for numerical data types such as integer or float.

If you need to use interpolating for an object or categorical data type, then first convert it to a numerical data type. For this, you need to encode your column first. The following piece of code will resolve your problem:

from sklearn.preprocessing import LabelEncoder 
from sklearn.preprocessing import OneHotEncoder

notes_encoder=LabelEncoder()

df['notes'] = notes_encoder.fit_transform(df['notes'])

After doing this, check the column's data type. It must be int. If it is categorical ,then change its type to int using the following code:

df['notes']=df['notes'].astype('int32')
0

For current users. After correcting proper data types.

df.groupby(['lat', 'lon']).apply(lambda group: group.interpolate(method='linear'))

Will work with Pandas version 2.0+ and is the best practice. Linear is the only supported method of interpolation for multi-index dataframes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.