3

Need to calculate each participating team's average goals per game (rolling_avg) up to a point in time. Since my database is highly normalized, this takes quite a lot of time (+8k rows)

Notes:

  • I'm counting the total goals (column 0) scored in a match, not each team's goals.

  • There are additional columns that are omitted here but could be relevant: one of them is a date_time column.

Example: in row #1 (the second row) we can see the first game of team 1249 (they're playing away). In this game, 3 goals were recorded. The next game of team 1249 takes place in row #10, and since that team is playing at home this time (as its id appears under home_team_id), I want the home_rolling_avg to be equal 3. It shouldn't take into account the current row.

Question:

How do I calculate each team's goals expanding average/mean, based on previous values in total_goals and excluding the current row, and assign this mean value to the relevant column (depends on whether the team is playing at home or away)?

    total_goals  home_team_id  away_team_id  home_goals  away_goals  home_rolling_avg  away_rolling_avg
0             2          1277          1241           1           1               NaN               NaN
1             3          1245          1249           2           1               NaN               NaN
2             1          1242          1246           0           1               NaN               NaN
3             4          1261          1248           1           3               NaN               NaN
4             2          1259          1240           2           0               NaN               NaN
5             3          2981          1268           1           2               NaN               NaN
6             1          1244          1255           1           0               NaN               NaN
7             1          1254          1276           1           0               NaN               NaN
8             7          1247         12140           5           2               NaN               NaN
9             3          5681          1270           2           1               NaN               NaN
10            1          1249          5681           0           1               NaN               NaN
# in line 10 above, 'home_rolling_avg' should show 3 (3/1)

Update:

a larger sample (n=100) can be found here. Added due a request by the kind piRSquared (answer here).

0

2 Answers 2

2
# easy tracking of long column names
hw = ['home_team_id', 'away_team_id']

# I found it easier to melt myself with some numpy help
hw_vals = df[hw].values.ravel()  # flatten 2 columns
idx_rep = df.index.values.repeat(2)  # repeat index [0, 0, 1, 1, ...
tot_rep = df.total_goals.values.repeat(2)  # repeat totals [2, 2, 3, 3, ...

# This is the reshaped series of team ids with total_goals
s = pd.Series(tot_rep, [idx_rep, hw_vals])

# groupby with a combination of expanding().mean() and shift()
e = s.groupby(level=1).apply(lambda x: x.expanding().mean().shift()).dropna()

# style preference of mine to do assignments using index values
# and to get it done in one line
df.set_index(hw[0], append=1).assign(home_rolling_avg=e).reset_index(hw[0]) \
  .set_index(hw[1], append=1).assign(away_rolling_avg=e).reset_index(hw[1])

enter image description here


Deeper Explanation

  • One of the main "tricks" of this question is to recognize the ids in two columns as a common id. We could use pd.melt, which I did. But I found the syntax to be uglier than what I ended up doing. And I know that numpy does it quicker anyway.
    • I have two columns of ids, I flatten it with ravel. This will double the length. In this example, length went from 10 to 20.
    • Then I create a new version of the existing index using repeat. Every value in the index gets repeated. Eg, [1, 2] becomes [1, 1, 2, 2]. I'll use this in conjunction withe the ids themselves to create a multi-index
    • repeat the total_goals column analogous to above
    • create a pandas series with the values of total_goals and a multi-index consisting of the prior index as the first level, and the team ids as the second level. The multi-index was specified by passing the list of arrays as the index parameter [idx_rep, hw_vals]
  • Now that I have this series, I can group by the second level of the index level=1 and do and expanding().mean().
    • However, I needed to execute this as a lambda in order to lag or shift it one period. The lag/shift was necessary to only account for the expanding mean up to but not including the current match.
  • We could use join, or a number of other techniques to get the relevant information combined with the original data set. However, this felt more natural to me.
    • By setting the to include the home team id, I can then assign to include a new column to a copy. I like this approach because it allows me to chain commands. So I do, by resetting the index and repeating the process for the away team. The assignment works because the indices line up naturally.

Alternative Approach
Using defaultdict + Counter from collections
Very similar to Steven Rouch

from collections import defaultdict, Counter
c, d = Counter(), defaultdict(int)
home_avgpg = pd.Series(index=df.index)
away_avgpg = pd.Series(index=df.index)

for row in df.itertuples():
    h = row.home_team_id
    a = row.away_team_id
    t = row.total_goals
    if h in c:
        home_avgpg.set_value(i, d[h] / c[h])
    if a in c:
        away_avgpg.set_value(i, d[a] / c[a])
    d[h] += t
    d[a] += t
    c.update([h, a])

@StevenRauch's answer is very fast.

0
1

I think this meets the problem statement. It uses itertuples to iterate each row and compute a running average:

teams_rolling_goals = {}
home_rolling_avg = []
away_rolling_avg = []

for row in df.itertuples():
    # get previous goal and game counts
    home_goals = teams_rolling_goals.get(row.home_team_id, (0, 0.))
    away_goals = teams_rolling_goals.get(row.away_team_id, (0, 0.))

    # calc a rolling average
    home_avg = np.nan if home_goals[1] == 0 \
        else home_goals[0] / home_goals[1]
    away_avg = np.nan if away_goals[1] == 0 \
        else away_goals[0] / away_goals[1]

    # save the averages for the row
    home_rolling_avg.append(home_avg)
    away_rolling_avg.append(away_avg)

    # accum rolling goals/games count
    teams_rolling_goals[row.home_team_id] = (
        home_goals[0] + int(row.total_goals), home_goals[1] + 1)
    teams_rolling_goals[row.away_team_id] = (
        away_goals[0] + int(row.total_goals), away_goals[1] + 1)

    print(row.home_team_id, home_rolling_avg[-1],
          row.away_team_id, away_rolling_avg[-1])

# save the results
df['home_rolling_avg'] = home_rolling_avg
df['away_rolling_avg'] = away_rolling_avg

Sample data used:

data = np.array([
    ('week', 'total_goals', 'home_team_id', 'away_team_id', 'home_goals',
     'away_goals', 'home_rolling_avg', 'away_rolling_avg'),
    (0, 2, 1277, 1241, 1, 1, np.nan, np.nan),
    (1, 3, 1245, 1249, 2, 1, np.nan, np.nan),
    (2, 1, 1242, 1246, 0, 1, np.nan, np.nan),
    (3, 4, 1261, 1248, 1, 3, np.nan, np.nan),
    (4, 2, 1259, 1240, 2, 0, np.nan, np.nan),
    (5, 3, 2981, 1268, 1, 2, np.nan, np.nan),
    (6, 1, 1244, 1255, 1, 0, np.nan, np.nan),
    (7, 1, 1254, 1276, 1, 0, np.nan, np.nan),
    (8, 7, 1247, 12140, 5, 2, np.nan, np.nan),
    (9, 3, 5681, 1270, 2, 1, np.nan, np.nan),
    (10, 1, 1249, 5681, 0, 1, np.nan, np.nan),
])

index = data[1:, 0]

df = pd.DataFrame(data=data[1:, 1:],
                  index=index,
                  columns=data[0, 1:])
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.