4

I have a Pandas dataframe with one column of numbers, similar to this:

id - val
0  - 100
1  - 200
2  - 100
3  - 400
4  - 300
5  - 100
etc

What I would like to do is to add a second column which is a list/numpy array of the values from the previous 3 rows:

id - val - val_list
0  - 100 - [] # Or [NaN, NaN, NaN]
1  - 200 - [100] # Or [NaN, NaN, 100]
2  - 100 - [100, 200] # Or [NaN, 100, 200]
3  - 400 - [100, 200, 100]
4  - 300 - [200, 100, 400]
5  - 100 - [100, 400, 300]
etc

Any idea how to solve this efficiently, preferably without looping?

3
  • How many rows does this data frame usually have?
    – Bhanu Tez
    Commented Oct 25, 2018 at 5:34
  • Order of NaN should be changed like [nan, nan, 100.0] ?
    – jezrael
    Commented Oct 25, 2018 at 5:40
  • Yes you are right @Jezrael Commented Oct 25, 2018 at 5:44

2 Answers 2

5

First I think working with lists in pandas is not good idea, if possible better is working with 2d numpy array here.

Use strides if performance is important:

N = 3
x = np.concatenate([[np.nan] * (N), df['val'].values])

def rolling_window(a, window):
    shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
    strides = a.strides + (a.strides[-1],)
    return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
arr = rolling_window(x, N)

df['val_list'] = arr[:-1].tolist()
print (df)
   id  val               val_list
0   0  100        [nan, nan, nan]
1   1  200      [nan, nan, 100.0]
2   2  100    [nan, 100.0, 200.0]
3   3  400  [100.0, 200.0, 100.0]
4   4  300  [200.0, 100.0, 400.0]
5   5  100  [100.0, 400.0, 300.0]
4
  • Downvoter, if there's something wrong with my answer, please let me know, so I can correct it. Thanks.
    – jezrael
    Commented Oct 25, 2018 at 5:44
  • Hey, thanks for the answer. Wasn't me who downvoted, give me a bit to check your solution Commented Oct 25, 2018 at 5:47
  • @FrancWeser - I know it, you have no reason.
    – jezrael
    Commented Oct 25, 2018 at 5:47
  • 2
    +1. Slightly more clear would be to have: shape = (a.size - window + 1, window), strides = a.strides*2.
    – kuzand
    Commented Oct 25, 2018 at 6:06
4

Hi use the following simple code.

df = pd.DataFrame([100,200,100,400,300,100],columns =['Val'])

temp = pd.concat([df.shift(3),df.shift(2),df.shift(1)],axis=1)
df['val_list'] = temp.apply(lambda x:x.tolist(),axis=1)
#
df = pd.DataFrame([100,200,100,400,300,100],columns =['Val'])
N=3
temp = pd.DataFrame()
for i in range(N,0,-1):
    temp = pd.concat([temp,df.shift(i)],axis=1)
df['val_list'] = temp.apply(lambda x:x.tolist(),axis=1)
6
  • Hi, please let me know while downvoting, so I can correct the code.
    – Bhanu Tez
    Commented Oct 25, 2018 at 5:50
  • Hi, thanks for the answer. Didn't downvote, but do you have an idea how to implement if the shift is not 3 but lets say 300 (so a list of the last 300 rows)? Don't want to write df.shift(300)...df.shift(1), thanks Commented Oct 25, 2018 at 5:53
  • Oh okay, if you need the last 300 rows.. definitely my implementation is of no use.. Thanks for clearing.. looks like today is my bad day.. so many people downvoting my answers.. I am just wondering why :(
    – Bhanu Tez
    Commented Oct 25, 2018 at 5:58
  • 2
    Hi, Franc.. wrote one more logic.. but with for loop(but you said you want to avoid for loop).. have a look.. thanks for upvote..
    – Bhanu Tez
    Commented Oct 25, 2018 at 6:09
  • 1
    Yep I'm trying to avoid loops where possible due to performance.. thanks!! Commented Oct 25, 2018 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.