21

shift converts my column from integer to float. It turns out that np.nan is float only. Is there a way to keep the shifted column as integers?

df = pd.DataFrame({"a":range(5)})
df['b'] = df['a'].shift(1)

df['a']
# 0    0
# 1    1
# 2    2
# 3    3
# 4    4
# Name: a, dtype: int64

df['b']

# 0   NaN
# 1     0
# 2     1
# 3     2
# 4     3
# Name: b, dtype: float64
1
  • 3
    you can use this hack: df['b'] = df['a'].shift(1).fillna(-1).astype(df.a.dtype) Jan 26, 2017 at 9:14

6 Answers 6

17

A solution for pandas under 0.24:

The problem is you get a NaN value that is float, so int is converted to float - see na type promotions.

One possible solution is convert NaN values to some value like 0 and then it is possible convert to int:

df = pd.DataFrame({"a":range(5)})
df['b'] = df['a'].shift(1).fillna(0).astype(int)
print (df)
   a  b
0  0  0
1  1  0
2  2  1
3  3  2
4  4  3

A solution for pandas 0.24+ - check Series.shift:

fill_value object, optional The scalar value to use for newly introduced missing values. the default depends on the dtype of self. For numeric data, np.nan is used. For datetime, timedelta, or period data, etc. NaT is used. For extension dtypes, self.dtype.na_value is used.

Changed in version 0.24.0.

df['b'] = df['a'].shift(fill_value=0)
3
  • Another issue is if the shift introduces a NaN thereby converting all integers to floats, there's some rounding that happens (e.g. on epoch timestamps) so even recasting it back to integer doesn't replicate what it was originally. Any way to fix this?
    – guy
    Jul 6, 2017 at 15:16
  • Unfortunately no.
    – jezrael
    Jul 6, 2017 at 16:00
  • @guy: It should be possible to use convert_dtypes function for this case as you can see in @totalhack answer (here). It should convert your data to so called ExtensionDtype which will stay integer but with support for pd.NA, so you can use shift method that will not convert your integers to floats as you have converted it to integer type with NA support.
    – Nerxis
    Apr 28, 2020 at 14:06
6

Another solution starting from pandas version 0.24.0: simply provide a value for the parameter fill_value:

df['b'] = df['a'].shift(1, fill_value=0)
1
4

As of pandas 1.0.0 I believe you have another option, which is to first use convert_dtypes. This converts the dataframe columns to dtypes that support pd.NA, avoiding the issues with NaN.

df = pd.DataFrame({"a":range(5)})
df = df.convert_dtypes()
df['b'] = df['a'].shift(1)

print(df['a'])
# 0    0
# 1    1
# 2    2
# 3    3
# 4    4
# Name: a, dtype: Int64

print(df['b'])
# 0    <NA>
# 1       0
# 2       1
# 3       2
# 4       3
# Name: b, dtype: Int64
4

You can construct a NumPy array by prepending a 0 to all but the last element of column a:

df.assign(b=np.append(0, df.a.values[:-1]))

   a  b
0  0  0
1  1  0
2  2  1
3  3  2
4  4  3
0

I don't like other answers which may change original dtypes. What if you have float or str in the data?

Since we don't need the first nan row, why not skip it?

I would keep all dtypes and cast back:

dt = df.dtypes
df = df.shift(1).iloc[1:].astype(dt)
0

Another solution is to use the replace() function and typecast:

df['b'] = df['a'].shift(1).replace(np.NaN,0).astype(int)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.