I've been meaning to document my ELO model for a while so here we go!
My model is ELO based with a few tweaks. We estimate teams' scores and then the margin using a combination of each teams' offensive strength, defensive strength and scoring accuracy.
Then turning the margin into a win probability involves considering the involved teams' historic variability in scoring to estimate the variance we expect in the predicted margin. We then combine the estimated margin and variance to estimate a probability using observations that AFL scoring tends to follow Brownian Motion which is discussed more
here.
Before getting into the nuts and bolts we'll start with some observations and assumptions.
Assumptions
- Good teams generate more shots and give away fewer shots than poor teams. Good teams also tend to generate
higher quality shots (closer to goal or at straighter angles or both). In the absense of expected score
data I'll be using a teams rolling scoring accuracy over some previous time period as a proxy for the
quality of shots a team generates.
- Mapping margin to win probability seems to be dynamic in that other modellers use different parameters in
their margin -> probability mapping function for different eras of football.
Instead, my algorithm will consider the uncertainty each
team brings to the table separately and convert expected margin to win probability dynamically for each
match.
So in my algorithm we may
predict the same margin for two matches, but we will be slightly more confident about a match between two
lower variance teams than two higher variance teams, and so our win probability will be closer to 50% for the
higher variance teams.
This means my algorithm updates as scoring trends change without the need to tune different parameters to
different eras of football.
I will probably do an entire blog post on this topic and explore the validity of it in more detail. Variance is interesting!
With these assumptions aside, here are the nuts and bolts:
Ratings and Predicting
Teams have 4 ratings. These are:
- Offence: Represents the number of shots a team can expect to produce (higher is better).
- Defence: Represents the number of shots a team can expect to give away (lower is better).
- Scoring Accuracy: \( \frac{\text{goals}}{\text{goals} + \text{behinds}} \) for the teams' last \(X\) matches.
- Uncertainty: Represents the standard deviation in the error of my predicted score (in points) for the last \(Y\) matches.
If I tend to me more wrong with respect to the score I predict for a particular team than other teams they'll have a higher uncertainty.
Historically this ranges from around 18 to 22 points and for good teams, lower uncertainty is better, but for bad teams higher uncertainty is better.
I.e. predictably good > unpredictably good > unpredictably bad > predictably bad.
\(X\) and \(Y\) are parameters we could tune, and ended up choosing \(X=22\) and \(Y=44\).
They weren't optimised too much, they represent 1 and 2 full home and away seasons in the current era respectively.
I can then use these ratings to estimate two metrics for each match:
- Margin prediction from the home teams perspective (\( \mu \)).
- Margin prediction standard deviation (\( \sigma\)).
Team scores are calculated as:
\[\text{homeScorePredict} = (\text{offenceHome} + \text{defenceAway} + \text{netHgaHome})\times(5\times\text{accuracyHome}+1) \]
\[\text{awayScorePredict} = (\text{offenceAway} + \text{defenceHome} + \text{netHgaAway})\times(5\times\text{accuracyAway}+1) \]
Where \(5\times\text{accuracy}+1\) represents the average number of points per scoring shot given a particular
accuracy, and hga represents an adjustment for home ground advantage in number of scoring shots. I'll explain how
that's calculated later.
Expected margin is then calculated as:
\[\mu = \text{homeScorePredict} -\text{awayScorePredict}\]
Standard deviation is calculated as:
\[ \sigma = \sqrt{\text{uncertaintyHome}^2 + \text{uncertaintyAway}^2 - 2\rho\cdot\text{uncertaintyHome}\cdot\text{uncertaintyAway}} \]
Essentially we expect variance in the final margin due to variance in the home teams score, variance in the away
teams score and covariance between the home and away teams scores.
Here \(\rho\) represents the correlation between home and away scores from the (at most) 2 years prior to the
current season (and is 0 for the first ever season). It turns out this standard deviation is around 35-40 for most
games which is close to the historical standard deviation in margins. This provides me with some good confirmation
bias ;).
I've plotted \(\rho\) by year below and with it the average total match score (home score + away score). In years of high scoring \(\rho\) tends to be around 0 or even positive. I.e. in these years if a particular team scored a lot, it had little effect on the scoring of their opponent. Or in the years with positive \(\rho\), a team was more likely to kick a higher score if their opponent also kicked a high score, strange!
This is in contrast to the current era where \(\rho\) is quite negative and so one team scoring highly means their opponent is more likely to score less. This all ties in with the discussion on scoring variance and is for another day.
Anyway, my estimate for the home teams win probability is then given by \(\Phi(\frac{\mu}{\sigma})\) where
\(\Phi(x)\) is the cumulative normal distribution function. There are more details about why we use this equation and it's connection with Brownian Motion
in
this blog post.
Home Ground Advantage
Now, to determine home ground advantage we use a proxy of travel distance + historic venue performance to estimate
the penalty or bonus each team receives due to venue effects, in terms of scoring shots.
The net travel penalty is calculated as:
\[\text{netTravelPenalty} = a \times ( (\text{Home Travel Distance in km})^c - (\text{Away Travel Distance in km})^c )\]
where \(a=-0.6, c=0.27\).
I found this is a pretty good proxy for most of the home ground advantage effect but doesn't give benefit to teams
like North Melbourne and Hawthorn playing in Tasmania. To make sure we give these teams an appropriate treatment
I've also considered a dynamic venue performance measure.
The venue performance (\(\text{venuePerformance}\)) is calculated in a few steps:
-
For the team of interest, what has been the historical effect of venue (call this \(\text{venueEffect}\))
at the venue of interest (over the last (max) 100 games) in terms of scoring shots, including adjustment
for travel distance?
\[\text{venueEffectHome} = \text{AVERAGE}(\text{shotsActualHome} - (\text{offenceHome} + \text{defenceAway}
+ \text{travelHome}))\]
-
How many matches has the team played at the venue? Call this \(n\). We cap \(n\) at 100.
-
Then the venue performance is given by \(\text{venuePerformance} = \text{venueEffect}\times f(n) \) where
\(f(n)\) is a weight we apply to the historical venue performance. For few games played we don't place much
weight on the historical performance but as more games are played we place more weight on past results.
\(f(n)\) was chosen so that \(f(n) = 0 \) for \(n\le 3\), \(f(4) \approx 0.03\), \(f(50) \approx 0.31\) and
\(f(100) \approx 0.52\). The function is exponential in nature and gets flatter with more games of
information.
-
Net venue performance for the match is then given by
\[\text{netVenuePerformance} = \text{venuePerformanceHome} - \text{venuePerformanceAway} \]
Basically we take into account the teams historical performance and adjust our scoring shot expectation in line
with this, and with more confidence when we have a larger sample of matches to base the calculation on.
So then the net HGA adjustment for the match is given by:
\[\text{netHga} = \text{netTravelPenalty} + \text{netVenuePerformance}\]
Then we adjust the home team and away team expected scoring shots according to:
\[\text{netHgaHome} = 0.5\times \text{netHga}\]
\[\text{netHgaAway} = -0.5\times \text{netHga}\]
Updating Ratings
In terms of updating the team rankings, after each match we calculate the error in scoring shots for each team:
\[\begin{align}
\text{errorHome} & = \text{shotsActualHome} - \text{shotsPredictHome} \\
& = \text{shotsActualHome} - (\text{offenceHome} + \text{defenceAway} + \text{hgaHome})
\end{align}
\]
\[\begin{align}
\text{errorAway} & = \text{shotsActualAway} - \text{shotsPredictAway} \\
& = \text{shotsActualAway} - (\text{offenceAway} + \text{defenceHome} + \text{hgaAway})
\end{align}
\]
We then allocate half of this error to the team who scored the shots' offence and half to the other teams' defence:
\[\text{errorHomeOffence} = \text{errorAwayDefence} = 0.5\times\text{errorHome}\]
\[\text{errorAwayOffence} = \text{errorHomeDefence} = 0.5\times\text{errorAway}\]
We then update the offence and defence rankings using the classic ELO method of: \(\text{New} = \text{Old} +
k\times\text{Error}\)
\[\text{offenceHome_new } = \text{offenceHome_old } + k\times\text{errorHomeOffence}\]
\[\text{offenceAway_new } = \text{offenceAway_old } + k\times\text{errorAwayOffence}\]
\[\text{defenceHome_new } = \text{defenceHome_old } + k\times\text{errorHomeDefence}\]
\[\text{defenceAway_new } = \text{defenceAway_old } + k\times\text{errorAwayDefence}\]
To update accuracy we just recalculate \( \frac{\text{goals}}{\text{goals} + \text{behinds}} \) for the latest 22
matches. And to update uncertainty we recalculate the standard deviation in the error in the teams expected score
(not shots) over the last 44 matches.
Now the \(k\) factor we use to update the offence and defence ratings changes depending on how much information we
have about a team in a given season.
This is according to the following equation:
\[k(m) = \frac{k_0}{m^b}\]
where \(m\) represents the number of games played (1 after the first match), \(k_0 = 0.30\) and \(b=0.18\). This
means \(k\) decays from 0.30 after the teams' first match of the season to about 0.17 at the end of the home and
away season.
Starting Values and Interseason Mean Reversion
In terms of starting values, we use an offence and defence of 10 for the very first season, accuracy of 0.4 and
uncertainty of 15.
We use uncertainty of 15 until there are enough games to calculate a standard deviation and we update accuracy
after the teams' first match.
For seasons where there is a new team, we given them an offence, defence and accuracy equal to the worst in the
league, and a uncertainty of 15.
At the end of each season we revert the offence and defence measures to the mean by 30%. We leave accuracy and
uncertainty untouched as they are based on the teams' previous 22 and 44 matches.
Performance
That's basically it, how does it go you ask?
Well it was used in the Monash tipping comp this year and conveniently it gives me numbers for every variety of
competition. It came 10th in the normal comp, 5th in the probabilistic one and 3rd in the gaussian one. The model
was tweaked throughout the year and in it's final form it would have done slightly worse than is reported on the
Monash site (but a bit better in other years).
The performance of this model by year is up
here.
Final Thoughts
Predictions for the next season are up
here too and they will be updated weekly following results as teams rankings change.
There are some extra steps in generating predictions for matches in future rounds. Essentially we add extra
variance for each team to account for potential changes in their rankings over each match before the future match
is played. This acts to bring matches closer to 50:50 the further they are away.
An example of this is the 2 matches between Richmond and Carlton. As of the start of season 2019 I have Richmond at 96% to beat Carlton
in Round 1 but only 87% in Round 21, while my margin prediction is 47 points for each game.
This will be explained more in a future post.
Also to come:
-
Comparing teams over time, who does my model rank as the GOAT?
-
Ranking dashboard to explore team rankings by year.