On Players Evaluation - Part III (Teams Elo)

Part I.

Part II.

Sherlock Holmes and Dr. Watson are camping in the countryside.

In the middle of the night Holmes wakes up Watson:

'Watson, what do you think these stars are telling us?

'Geez, Holmes, I don't know, maybe it's going to be a nice weather tomorrow?

'Elementary, Watson! They are telling us our tent has been stolen!

Iconic Soviet joke.

Estimating a hockey player via Elo ratings is a highly complex task. Therefore, we shall wield the dialectic approach of getting from the simpler to the more complicated, and will tackle a seemingly simplistic task first. Let's work out the Elo ratings for the NHL teams as a whole first. After all, it's the teams who compete against each other, and the outcome of this competition is a straightforward result.

So, let's examine a match between Team A and Team B. They have ratings R_a and R_b. These ratings, or, more precisely, their difference R_a-R_b, defines the expected results E_a and E_b on the scale from 0 to 1. The teams play, one wins (S=1), another loses (S=0). To adapt this to the Elo scale, let's consider win 1 point, loss 0 point. The new ratings R_a^' and R_b^' will be (K is the volatility coefficient):

Outcome	Sa	Sb	Sa-Ea	Sb-Eb	dRa	dRb	Ra'	Rb'
Team A Wins	1	0	1-Ea	-Eb	K-K*Ea	-K*Eb	Ra+K-K*Ea	Rb-K*Eb
Team B Wins	0	1	-Ea	1-Eb	-K*Ea	K-K*Eb	Ra-K*Ea	Rb+K-K*Eb

and the teams are ready for usage in the next meeting with their new ratings R_a^' and R_b^', reciprocally.

'Wait!', will ask the attentive reader, 'Not all possible outcomes are listed above! What about the OT/SO wins where both teams get some points.' And he will be correct. In these cases we must admit that the loser team scores 0.5 points, so unlike a chess game where the sum of the results is always 1, in the NHL hockey the total sum of results varies and can be either 1 or 1.5. Note, were the scoring system 3-2-1-0, then we could scale the scores by 3 rather than by two and get the range 1-⅔-⅓-0 where every result sums to 1. Alas, with the existing system we must swallow the ugly fact that the total result may exceed 1, and as the result the ratings get inflated. Which is a bad thing, sure.

Or is it? Remember, the Elo expectation function only cares about the differences between ratings, not their absolute values. And all teams' ratings get inflated, so all absolute values shift up from where they would've been without the loser's point. Whom would it really hurt? The new teams. Naturally, we must assign an initial rating to every team at the starting point. One way could be assigning the average rating of the previous season to the new team. But we prefer a different and a much more comprehensive solution. We claim that since the teams that at the start of the next season are different enough beasts from those that ended the previous ones, so that the Elo ratings should not carry over from season to season at all! Therefore all the teams start each season with a clean plate and an identical Elo rating R_o.

Once again, the attentive reader might argue, 'What about mid-season trades and other movements?' Well, dear reader, now you have a tool to evaluate impact of the moves on the team. If there is a visible tendency change, you can quite safely associate it with that move. Overall, the 82 game span is huge to soften any bends and curves in the progression of the Elo ratings along the season.

Speaking of game spans, we must note one more refinement being done to the ratings. In the chess world, the ratings of the participants are not updated throughout the length of the event, which is usually 3-11 games. The ratings of the participants are deemed constant for the calculation of rating changes, which accumulate, and the accumulation is actually the rating change of each participant. We apply a similar technique for the teams' Elo calculations: we accumulate the changes for the ratings for 5 games for each team and "commit" the changes after the five-game span. The remainder of the games is committed regardless of its length, from 1 to 5. Why 5? We tried all kinds of spans, and 5 gave the smoothest look and the best projections.

Now, as a demonstration, let's show how we calculate the possible rating changes in the much anticipated game where Minnesota Wild is hosting Columbus Blue Jackets on December, 31st, 2016:

R_cbj = 2250, R_min = 2196, E_cbj = 0.577, E_min = 0.423, K = 32 (standard USCF).

Outcome	Scbj	Smin	S-Ecbj	S-Emin	dRa	dRb	Ra'	Rb'
CBJ W Reg	1	0	0.423	-0.423	+13.53	-13.53	2263.53	2182.47
CBJ W OT	1	0.5	0.423	0.077	+13.53	+2.47	2263.53	2198.47
MIN W OT	0.5	1	-0.077	0.577	-2.47	+18.47	2247.53	2214.47
MIN W Reg	0	1	-0.577	0.577	-18.47	+18.47	2231.53	2214.47

Note: MIN gains rating when it gets a loser's point.

Here is a dynamic of Elo changes (without five game accumulation) for the Metropolitan Division, as an example.

See more detailed tables on our website: http://morehockeystats.com/teams/elo

Ok, we got the ratings, we got the expected results, can we get something more out of it?

To be continued...

Happy New Year to everyone!

On Players Evaluation - Part II (Elo)

Part I.

The Elo rating system is the system used for evaluation and comparison of competitors. Up until today it's been mostly applied in the domain of board games, most well-known in chess, but also in disciplines such as draughts or go. The Elo system, named after its inventor, Prof. Arpad Elo, who first published it in the 1950s in the US, is capable to produce a reliable score expectation for an encounter between two competitors who oppose each other.

For those who are not familiar with chess or draughts, let's take a look on how the Elo ratings work:

1) In an encounter between two competitors, A and B, assume they have ratings R_a and R_b.

2) There is a function that maps the expected result for each player given the opponent:

E_a = F(R_a, R_b)

E_b = F(R_b, R_a)

where F is a monotonic non-decreasing function bounded between minimum and maximum possible scores, such as 0 and 1 in chess. An example for such a function would be arctan(x)/π + 0.5 .

E_a+E_b should be equal to maximum possible score.

In practice a non-analytical table-defined function is used that relates only on the difference between Ra and Rb, and not their actual values. The function can be reliably approximated by the following expression:

E = 1 / [ 1 + 10^(R_b-R_a) / 400 ]

which works well with ratings in low 4-digit numbers and rating changes per game in 0-20 range.

3) After the encounter, when real scores S_a and S_b have been registered, the ratings are adjusted:

R_a1 = R_a + K*(S_a-E_a)

R_b1 = R_b + K*(S_b-E_b)

Where K is a volatility coefficient, which is usually higher for participants with shorter history, but ideally it should be equal for both participants. The new ratings are used to produce the new expected results and so on.

The Elo rating has several highly important properties:

1) It gravitates to the center. As rating R of a participant climbs higher, so does the expected result E, which becomes difficult to maintain, and a failure to maintain it usually results in a bigger drop in the rating.

2) It's approximately distributive. If we gather N performances and average the opponents as R_av, the expected average performance as E_av = F(R_a, R_av), and the actual performance as S_av, then the new rating R_aN' = R_a + N*K*(S_av-E_av) will be relatively close to R_aN obtained via direct R_a reciprocal update after each of the N games.

3) It reflects tendencies, but overall performance still trumps it. Given the three players with ten encounters against other players with the same rating, when the performances are (W - win, L - loss):

For player 1: L,L,L,L,L,W,W,W,W,W

For player 2: L,W,L,W,L,W,L,W,L,W

For player 3: W,W,W,W,W,L,L,L,L,L

player 1 will end up with the highest rating of the three, player 2 will be in the middle, and player 3 will have the lowest one - but not by a very big margin. Only when the streaks become really long the Elo of a lower performance may overcome the Elo of a higher one.

And how does Elo stack against the four Brits?

* Goodhart's Law: pass. It measures the same thing it indicates.
* Granger's Causality: pass. It is a consequence of a performance by definition, and a prediction of future peformance, by definition.
* Occam's Razor: pass. The ratings revolve around the same parameter they measure.
* Popper's Falsifiability: partial pass. The predictions of Elo sometimes fail, because they are probabilistic. However, the test of time and the wide acceptance indicate that the confidence level holds. Elo was even used for "paleostatistics" when the ratings were calculated backwards until middle XIX century, and the resulting calculations are well-received by the chess historians' community.

The only well-known drawback of Elo is the avoidance by top chess players of competition against much weaker oppositions, especially when facing White, as such a game can be drawn relatively easily by the opponent, and the Elo rating of the top player could take a significant hit resulting in a drop of several places in the rating list.

Now, to the question of the chicken and the egg - where do the initial Elo ratings come from? Well, they can be set to an arbitrary value of low 4-digit number. Currently a FIDE beginner starts with the rating of 1300. If the newcomer is recognized as being more skilled than a beginner, then a higher rating is assigned based on rating grades for each skill level, sort of an historical average of the newcomer's peers.

And... What does all this have to do with hockey?

To be continued...

On Player Evaluation Systems - Part I

In the previous post we mentioned the Goodhart's Law and how it threatens any evaluation of an object. We said that it traps the Corsi/Fenwick approach because it substitutes the complex function of evaluation of a hockey player by a remarkably simple stat - shots.

Goodhart's law is not alone. In any research it is preceded by the two pillars: Popper's law of falsifiability and the Occam's razor. A theory willing to bear any scientific value must comply with both, i.e. produce hypotheses that can be overthrown by experiment or observation (and then relegated to the trashcan), and must avoid introduction of new parameters beyond the already existing ones. Add Granger causality into the mix and we see that the four Brits presented the hockey analytics society with pretty tough questions that the society - at least the public one - seems to be trying to avoid.

The avoidance will not help. Any evaluation system will not be able to claim credibility unless it complies with the four postulates above, and within that compliance issues measurable projections.

To be continued...

On The NHL Scoring System (Part II)

Part I.

Goodhart's law is the bane, the safeguard and the watchdog of everyone who tries to make conclusions from sample data. The "Schroedinger Cat of Social Sciences" practically says, if you want people to do X, but you reward them for doing Y, they will be doing Y rather than X. We start seeing that in the "possession analytics", based on shots taken, that the players begin to shoot from everywhere to get their possession ratings up. But we digress - the topic is the scoring system, we'll save that note for another blog entry.

We want the NHL hockey to be spectacular. That's the main objective (beyond being fair and competitive, otherwise look for Harlem Globetrotters). In the past the spectacular was fighting as much scoring and winning games, but that taste of the public changed, and the fighting went away. It was not directly related to scoring and winning, it was just an extra free show provided.

Now we're left with scoring and winning. These two are closely tied, and not necessarily as a positive feedback, since not allowing your opponent to score also helps winning. A 2-1 win is practically just as valuable as a 7-2. So in the mid-2000s, the winning objective, the points objective took over the scoring objective. And from the previous post we see that the existing 2-2-1-0 points system encourages low-intensity game preferably slipping into the OT. On the other hand we also noted that the 3-2-1-0 points system would encourage teams to clamp down and protect their minuscule leads. Looks like a circle to break...

Well, here comes Goodhart's law. You want teams to score, or at least try to score, but you reward them for achieving points. So what they do is concentrate on getting points. Therefore, if the NHL want to see score-oriented hockey, the NHL needs to reward scoring, and not points. Still, the points have been used to determine playoff spots, so something has to give.

First, let's take a wild ride by suggesting that we rank teams by the amount of goals scored. That would lead to a pretty drastic change and the end of hockey as we know it. This will lead to situations where a team might play for a period without a goaltender in a playoff race. In general, the goaltending position will deteriorate, and aren't we loving the spectacular saves just as we love slick goals? Probably, that would be too much.

Thus, we can mitigate to allow the goal differential to be the ranking criteria. At the end of the season, the teams with the higher goal differentials will be ranked at the top, and the wins-ties-losses, well, they get relegated to tie-breaks. The incentive to score rather than to hold the opponent increases, because while now the competitors in their games cannot score more than two points, they still can score a bigger goal differential! All the lazy skating to finish the game after it's 4-0 or 5-1? Gone.

This idea is actually not novel. It's been used for a long while in team chess tournaments. Such tournaments consist of matches, where each player of one team plays against an opponent of the opposing team at the same time. Each player's individual score (win, draw or loss) is accumulated into the total score. So a match of 8-player teams, where one team had 5 wins, 1 draw and two losses ends up with 5.5-2.5 score, essentially "the goal differential". At the end of the tournament, the scores accumulate, and the teams are ranked according to them. You can see the crosstables of historical chess tournaments at the wonderful Olimpbase website.

And if you feel that the fact of winning or losing the game should be have more weight than just a tie-break (by the way, there will be less tie-breaks on goal differential), that is easy to factor in, just add a bonus "goal" to the winner, like it is done in the shootout now. Better, add two bonus "goals" for winning in regulation, one bonus "goal" for winning in the OT, abolish shootouts.

On The NHL Scoring System (Part I)

There was nothing wrong with ties. The 2-1-0 point system works fine in various sports around the world. It's just ... not fitting into the mind of a North American sports fan. "Who won?" - "It was a tie." - "Who won on a tiebreak?" Basketball and baseball do not have ties, and American Football has them at a rate of 1-2 times per whole season. So more than ten years ago NHL went with the flow and abolished ties, introducing the shootout, and with a twist, where the team making it past the regulation would still get the point, and a 2-2-1-0 point system came to life.

Since then the argument rages, whether the ties should come back, or whether the consolation point should be taken away, or whether the much more energetic 3-2-1-0 point system, adopted across the ocean and by the IIHF should make its way into the NHL as well. The feeling that there is something unhealthy when a team loses and still gets something, while the winner is not penalized is nagging.

The argument from the NHL leadership claims the system creates denser standings and thus more interest and drama throughout the season is a valid one. However, this system, as we show below, creates a wrong incentive.

The standings in the NHL are defined by a points total, and the seeding in the playoffs are first and foremost the divisional standings. The relative standings across conferences have a rather minor effect of the potential home advantage in the Stanley Cup Finals, the same standings within the same conference but across divisions have an impact on the seedings in the whole playoffs, but also to a limited effect. Therefore, at least with the exception of intradivisional games, but possibly including these games too (especially against the competition that has fallen out of the playoff picture), the only thing that matters are the points accrued by the team itself, and not the points the opposition gathers. Let's wield the statistic that says that 25% of the games go to the overtime and the

So what are the point expectations in a 2-2-1-0 system? Let's compare a few situations when teams A and B play.

1. Team A has 75% chance of winning the game (that's a huge, possibly maximum imaginable favorite odds)
2. Team A has 67% chance of winning.
3. Team A has 60% chance of winning.
4. Team A has 50% chance of winning.

Let's wield the statistic that says that 25% of all games go to the overtime and the shootout occurs in 40% of these games. Let's also assume that the 3-vs-3 overtime is more random and reduces by half the advantage of the better team (i.e. 75-25 becomes 62.5-37.5), and that the shootout is completely random, so the chances of winning it are 50/50. Then, the probabilities of the outcome become:

	Chance	PwReg	PwOT	PwSO	xPoints
Team A	75%	0.5625	0.09375	0.05	1.51875
Team B	25%	0.1875	0.05625	0.05	0.73125
Team A	67%	0.5025	0.08775	0.05	1.39275
Team B	33%	0.2475	0.06225	0.05	0.85725
Team A	60%	0.45	0.0825	0.05	1.2825
Team B	40%	0.3	0.0675	0.05	0.9675
Team A	50%	0.375	0.075	0.05	1.125
Team B	50%	0.375	0.075	0.05	1.125

Now let's consider than the stronger team A plays intentionally for overtime and manages to force it in 75% of the cases.

	Chance	PwReg	PwOT	PwSO	xPoints
Team A	75%	0.1875	0.28125	0.15	1.55625
Team B	25%	0.0625	0.16875	0.15	1.19375
Team A	67%	0.1675	0.26325	0.15	1.49825
Team B	33%	0.0825	0.18675	0.15	1.25175
Team A	60%	0.15	0.2475	0.15	1.4475
Team B	40%	0.1	0.2025	0.15	1.3025
Team A	50%	0.125	0.225	0.15	1.375
Team B	50%	0.125	0.225	0.05	1.375

In ALL cases it's worth for both teams to steer the game into OT. For the even odds case, the expectation gain is a whopping 0.25 points! Even in the case of super, uber favorite, it's still worth for that team to head to overtime, as it projects a gain of 0.04 points. And the gains for the underdogs are so big that there is no reason for the underdog to disturb the force of the overtime, so they will happily comply! Meaning: we'll see more fun overtime, we'll see more dumb shootouts, but more importantly the 60 minutes of hockey will lose a lot of their significance. The only quantative incentive to finish the game in regulation becomes denying extra points for your opponents - hardly a significant matter in what, fifty out of the eighty-two season games!

Now, let's repeat these calculations with 3-2-1-0 point system and combine them into another table:

		2-2-1-0			3-2-1-0
	Chance	Exp25%OT	Exp75%OT	Δexp	Exp25%OT	Exp75%OT	Δexp
Team A	75%	1.51875	1.55625	+0.0375	2.08125	1.74375	-0.3375
Team B	25%	0.73125	1.19375	+0.4625	0.91875	1.25625	+0.3375
Team A	67%	1.39275	1.49825	+0.1055	1.89525	1.66575	-0.2295
Team B	33%	0.85725	1.25175	+0.3945	1.10475	1.33425	+0.2295
Team A	60%	1.2825	1.4475	+0.165	1.7325	1.5975	-0.135
Team B	40%	0.9675	1.3025	+0.335	1.2675	1.4025	+0.135
Team A	50%	1.125	1.375	+0.25	1.5	1.5	0
Team B	50%	1.125	1.375	+0.25	1.5	1.5	0

Now there is no incentive for the stronger team to push for overtime, and even the gain for the weaker team decreased. 3-2-1-0 definitely encourages a regulation decision!

Reasons where brought up against the 3-2-1-0 system. One states that the spread over the standings will be too thin, and more teams will be eliminated from the playoff race early. This argument has had no statistical support, and the element of drama when a team pulls a goalie in a tied score trying to force a 3-0 point win may actually more than make up for it. Another argument refers to soccer studies that claim the 3-1-0 point system there encourages teams to sit on their early leads trying to stifle the game, which decreases the attractiveness of the game. This argument is more valid, although it's notably harder to preserve a lead in hockey than in soccer. But beyond that this argument prompts for another, a truly revolutionary suggestion...

To be continued.

Talent, Skill and the NHL

This wasn't supposed to be the first blog post here, but a twitter exchange prompted me to write this one.

On Talent In General

When you want to do some useful work, you need a skill to do that work. Naturally, one doesn't need a skill to tweet, but that's not a useful work to start with. But to do stuff that actually profits you a certain level of skill is absolutely necessary.

In order to have the skill, you need to learn it, and then to improve it. And there are only two basic factors that define how well you learn and improve in the skill - the talent and the effort. The bigger is your talent, the bonus from the nature, whether it's thanks to inborn memory, flexibility, or a quick eye, the less effort you need to achieve the given level of skill. And the trade-off is not even linear, there are areas, mostly creative ones, such as music or painting where no amount effort, grit and determination can bring you to a certain level of skill.

On the other hand, the bigger the talent, the less necessary the effort becomes, and at the extreme level of talent, also known as ... genius the person sometimes doesn't need practically any effort to improve at an incredible pace. This phenomena, already extremely rare, is mostly restricted to mind activities, bound by the necessity in constant exercise to maintain a high level of skill in a physical activity. Names of Wolfgang Amadeus Mozart in composing, Jose Raul Capablanca in chess or Robertino Loreti in music come to mind when we talk about such geniuses. Mozart was composing himself already at age five, Capablanca learned the game of chess from observation only, but won the Cuban Championship when he was twelve, and Loreti became a European super-star shortly after he was noticed singing folk tunes on the streets of Rome.

Talent And Skill In Hockey

Hockey is also a work that requires skill. It's a complex skill that consists of many abilities: skating, observation, agility, strength, endurance, wit an others. Since it's a team game, the team consists of players that excel at these abilities on a different level, and a "complete hockey player" would actually be someone who can skate like Mike Gartner, is observant like Wayne Gretzky, can shoot like Mike Bossy, has the strength of Eric Lindros, endurance of Nicklas Lidstrom, and, actually, can easily take a hit from Cam Neely (and hit like Neely, too) and hold his ground in a fight against Tie Domi, and such a "complete hockey player" would exist mostly in the computer games.

Nevertheless, of course the hockey players have different levels of ability in these dimensions of the hockey skill, and, unfortunately, today, mostly the ability to stickhandle is exclusively classified as 'skill'. No, the hockey skill is composite, and the wonderful dekes are just one aspect of it. Naturally, the most appealing, and probably the most important dimensions (we're not talking about goaltending here, but the reader can make similar projections to that position, too) are the ones directly relating to the goal scoring, and the players who excel at them are generally valued higher. However, in the way the hockey is defined by the NHL rules and and the NHL tradition, other qualities of the skill - hitting, blocking shots, fighting are required to make the complete hockey team. In a different league, such as the USSR league was, with very limited hitting and explicitly prohibited fighting, the sportsmen would develop more into the goal-scoring oriented hockey players.

Where The Talent Comes From

Well, from mothers' wombs. But then, the players usually begin to learn the skill of hockey from a very young age, and by the age of the NHL draft eligibility, their talent is well-evaluated and the positions in the draft order give a good approximation of the order of the talent of the available players. There are few exceptions, and these mostly are the European players, especially the Eastern-European ones who do not take part in the draft, but continue to develop in their leagues, such as the KHL. But the rule that the biggest chunk of the hockey talent is available at the annual draft, and that the talent is sorted according to the actual draft picks pretty much holds.

Therefore, the teams that feature the higher draft picks in their roster are on average definitely more talented ones than the ones with the lower picks. Are they most skillful? Not necessarily. Remember, that in the first part of this essay we stated that to develop a skill, both talent and effort are required. Some players, for whatever reason, fail to put the necessary amount of effort to achieve the skill level expected for their talent, and became disappointments, or even draft busts. Some, on the other hand, put a great effort and determination, and leap beyond such expectations. The latter ones, unfortunately, are bound by that aforementioned ceiling that sometimes lack of talent produces.

A team whose top draft picks underperform on a regular basis must recognize it has a culture problem. When time after time, players, who are supposed to be easy learners and advance rapidly, stall or degenerate it means that the organization, and, pardon the pun, it's farm, has a soil problem, that even the best seeds planted in it fail to yield the desired fruit.

We tried to quantize the talent usage by the NHL teams and coaches.