I have two separate dataframes representing different types of time-based data. One contains hundreds of thousands of timestamps spread across several months. This dataframe has columns representing the month of the year, the time of the day, and the measured temperature. The second dataframe consists of replacement temperatures for each month/hour combination. The data looks roughly as follows:
df1
Timestamp | Month | Hour | Temperature |
---|---|---|---|
1/1/2021 00:00:00 | 1 | 0 | 10 |
1/1/2021 00:00:05 | 1 | 0 | 11 |
1/1/2021 00:00:07 | 1 | 0 | 8 |
1/1/2021 00:00:15 | 1 | 0 | 12 |
1/1/2021 00:01:00 | 1 | 1 | 13 |
etc.
df2
Hour | Jan | Feb | Mar | Apr | etc |
---|---|---|---|---|---|
0 | 9 | 12 | 10 | 12 | etc |
1 | 10 | 11 | 14 | 15 | etc |
2 | 8 | 7 | 12 | 16 | etc |
df2 contains a row for each hour of the day, and a column for each month of the year (In the real data set the months are numeric, I wrote the names to make the description clear).
I need to map the data contained for the month/hour in df2 to the Temperature column in df 1. So df1, after editing, should appear as follows.
New df1
Timestamp | Month | Hour | Temperature |
---|---|---|---|
1/1/2021 00:00:00 | 1 | 0 | 9 |
1/1/2021 00:00:05 | 1 | 0 | 9 |
1/1/2021 00:00:07 | 1 | 0 | 9 |
1/1/2021 00:00:15 | 1 | 0 | 9 |
1/1/2021 00:01:00 | 1 | 1 | 10 |
I have made it work using a nested for loop, as follows:
for month in df2.columns:
for hour in df2.index:
dT = df2.loc[hour, month]
df1.loc[(df1['Month'] == month) & (df1['Hour'] == hour), 'Temperature'] = dT
The nested for loop cycles through all months and hours in df2, locates the cells in df1 with the matching month and hour, then sets the temperature equal to the temperature read out of df2.
But this code is both hard to read and super slow to execute. Does somebody know a better way??
Thanks!