5

My terminology is horrible so this one deserves some explanation. Imagine that I have a DataFrame like this (which I call the "long" table):

time       stock     price
---------------------------
13:03:00   AAPL      100.00
13:03:00   SPY       200.00
13:03:01   AAPL      100.01
13:03:02   SPY       200.01
13:03:03   SPY       200.02
.
.
.

and I wanted to convert it to a DataFrame like this (which I call the "wide and sparse" table):

time       AAPL      SPY
---------------------------
13:03:00   100.00    200.00
13:03:01   100.01    Nan
13:03:02   Nan       200.01
13:03:03   Nan       200.02

So obviously this is quite a transformation. Is there a built-in function that does this? It seems like it might be a pretty common thing to want to do.

Thanks!

1 Answer 1

6

You can use pivot:

df = df.pivot(index='time', columns='stock', values='price')
print (df)
stock       AAPL     SPY
time                    
13:03:00  100.00  200.00
13:03:01  100.01     NaN
13:03:02     NaN  200.01
13:03:03     NaN  200.02

Another solution with unstack:

df = df.set_index(['time', 'stock']).price.unstack()
print (df)
stock       AAPL     SPY
time                    
13:03:00  100.00  200.00
13:03:01  100.01     NaN
13:03:02     NaN  200.01
13:03:03     NaN  200.02

But if get:

ValueError: Index contains duplicate entries, cannot reshape

Is necessery use pivot_table with some aggregate function, default np.mean.

print (df)
       time stock   price
0  13:03:00  AAPL  100.00
1  13:03:00   SPY  200.00
2  13:03:01  AAPL  100.01
3  13:03:02   SPY  200.01
4  13:03:03   SPY  200.02
5  13:03:03   SPY  500.02 <- duplicates for same time and stock 


df = df.pivot_table(index='time', columns='stock', values='price')
print (df)
stock       AAPL     SPY
time                    
13:03:00  100.00  200.00
13:03:01  100.01     NaN
13:03:02     NaN  200.01
13:03:03     NaN  350.02

Another possible solution for duplicates time and stock:

df = df.groupby(['time', 'stock']).price.mean().unstack()
print (df)
stock       AAPL     SPY
time                    
13:03:00  100.00  200.00
13:03:01  100.01     NaN
13:03:02     NaN  200.01
13:03:03     NaN  350.02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.