A Geek's Eye View of the European Ladies Championship

How wrong he was.

The recent European Championships were widely criticized for the results in the ladies and ice dancing events.  The ISU responded by issuing a press released that basically said "It's all good."  According to this press release, everyone is happy with the system.  There are no complaints.  And all that is needed is just a little more continuing education for the judges to make a perfect system even more perfect.

Excuse me? Were the sources of this press release watching the same coverage, reading the same reports, surfing the same internet as the rest of us?  In the ladies event, many expressed the view that Irina Slutskaya was shamelessly held up in the program components to win, and others that review of video from the free skating shows that Julia Sebestyen was robbed of 4.5 points, and the silver medal, due to an error by the callers in designating a poor triple Lutz a double.  These criticisms were expressed not just by fans (who were burning up the internet worldwide), but also by former champions and ISU judges.  No complaints?  Not!

Contrary to the head-in-the-sand view from the ISU,  it's not all good.  Regardless of whether Slutskaya was actually held up or not, the reactions to Europeans make clear that the ISU still has serious credibility problems, that more and more people recognize that the judging system does not prevent judges from holding up (or pushing down) skaters if they feel like it, and that the possibility of "protocol" judging is not eliminated by the judging system. Increasingly, people recognize that the callers can make mistakes that can affect the podium, and that their decisions are indeed part subjective judgments open to question (such as whether a jump is truly cheated, or a spin position held long enough to count, etc.).

Given all this controversy, it seemed like it might be interesting (at least to a geek) to look at the scores from the ladies event and see what they might tell us about the event and the judging process.  That is what we did, and this is what we found.

Ladies Championship, Event Results

Before even beginning the calculations one thing was immediately noticed.  In the ladies event there were two cases of judges entering the wrong scores (identified by entering marks of less than one).  Elena Liashenko was a victim of this in the free skating, and to Jaqueline Belenyesiova in the short program.  This error also occurred in other events at Europeans.  One can only guess how many other marks are slightly off due to entry errors, since only the marks of less than one glaringly stand out, but it is reasonable to expect there are some.

For Liashenko, the eighth set of marks in the free skating protocol have an entry error.  Marks from columns 3, 5, and 7 were the discard marks, so the marks from column 8 were part of the results.  Due to the entry error, Liashenko lost 0.06 points.  Not many points 0.06, and it didn't affect the medal result this time; but if this defect of the system is not corrected eventually an entry error like this will affect a medal result, at which point there will no doubt be hell to pay.  But the ISU stubbornly refuses to add the simple "idiot check" that any competent programmer or system engineer would include in the user interface.

The following is the order of finish for the top six skaters.  For comparison we also show the Total Factored Place using the former method for combining short and long program results.

Since the system does not have the mathematical ability to distinguish between closely scored skaters, any skaters within about 1.5 points of each other are statistically tied.  The order of finish for the long program in this event is statistically meaningless for the top three ladies.  They are effectively tied.  The final placements for Poykio and Liashenko are also statistically meaningless, (since the system does not have the mathematical fidelity to truly distinguish between results that differ by only 0.91 points) as are the places for many of the other skaters.  A calculation of the statistical uncertainty in the total scores for this event shows that 14 of 37 places (39%) are not statistically significant -- meaning 14 skaters were effectively tied with another in this event.

Note also in the above table, that a 4.5 point error in Sebestyen's score moves her up to second place overall, and the silver medal.  The correctness of one decision of the callers has a profound effect on the results in this event.  Given the mathematical properties of the system it was previously predicted that the incorrect identification of a double vs. triple would typically move a skater one to two places in the results.  In this event it proved to be two places.

Distribution of GoEs

The distribution of GoEs in the scoring is another area that came into criticism after Europeans.  The dearth of GoEs of +2 and +3 is obvious from simple inspection of the protocols.  The following table gives the detailed distribution of the GoEs for the ladies short and long programs.

DISTRIBUTION OF GoEs FOR 2005 EUROPEAN CHAMPIONSHIPS LADIES EVENT

Out of 8016 GoEs, 85% lie in the range -1 through +1, and most of those (2/3) are 0s.  GoEs of +2 and +3 are virtually non-existent (less than 1%).

This distribution of marks strains credibility.  These are among the best skaters in the World, only a few of the top ladies are missing, and no one can do an element better than +1!  Hard to believe.

Using triple jumps as an example, a GoE of +1 gives a skater about 20% more points than the base value for the average triple jump.  That would seem to say that a +1 GoE should correspond to a jump that is only 20% better than "just ok."  One would think that an element only 20% better than just ok would occur more than 14% of the time.  Similarly, a GoE of +2 is only about 40% above the base value.  One would think that an element only 40% better than just ok would occur more than 0.7% of the time.  But that is not the way the marks are assigned.

It appears there is a major disconnect between the judging standard and the actual points to be won.  It is as though in the decathlon the top points for the pole vault were set for a vault of 40 feet, knowing full well the best an athlete can do is about 20 feet.  The points are out there to be earned, but no one is ever going to earn them.

So, what then is the point in having a GoE of +3 if the judges are never going to use it?  Or even +2 for that matter?

The standards for the top GoEs make no sense from a practical point of view.  It is suggested that the standard for a +3 GoE should be revised so that the top 1-2% of the executed elements have a chance of earning that score, with 4-5% for +2 and no more than 50% for 0.  This way, the standard is tied to the actual range of skill for each element as they are performed today, not some abstract standard the skaters will never reach.

The fact the GoEs are not realistically spread out is one of the reasons the judging system has such a hard time mathematically separating nearly equivalent performances.  In any evaluation process, the quality of the evaluation comes down to what (in science and engineering) is known as the signal-to-noise ratio.  To get a good evaluation you want a big signal and low noise.  Restricting the range of the GoEs so unnaturally decreases the signal, while decreasing the number of judges (by randomly throwing away judges) increases the noise (uncertainty).  The worst of both worlds.

To improve the quality of competition results the system needs to increase the signal (spread out the marks) and reduce the noise (eliminate random selection of marks).  Without some action along these lines, one can expect that 1/3 or more of the places in competition will continue to be statistically meaningless.

And who cares if the results are statistically meaningless?  The skaters, one would hope; for the more statistically meaningless the results, the more likely random selection of the judges will skew the results away from the true opinion of the panel as a whole, and contaminate the results and the medal decision.

Distribution of GoEs for Individual Elements

One can go a step farther in studying the GoEs and look at their distribution for individual elements.  For example, how did the judges score all the double Axels in the ladies event?

Double Axels were scored 33 times in the Short Program and 18 times in the Free Skating  The following table gives the distribution of GoEs for the two programs, and the percentage of the time each GoE was given.

DISTRIBUTION OF GoEs FOR DOUBLE AXELS

      
Amazing!  The best skaters in the world and no one can do a +2 or +3 double Axel.  No one!

Well maybe a double Axel is too difficult a jump, after all they have only been doing that jump since they were about 12 years old.  Let's look at something really easy, like a level 1 straight line step sequence.  No difficulty to this element required at all.  The following table is the distribution of GoEs for the two programs, with this element scored 15 times in the Short Program and 12 times in the Free Skating.

DISTRIBUTION OF GoEs FOR STRAIGHT LINE STEP SEQUENCE LEVEL 1

 
Amazing!  The best skaters in the world and no one can do the simplest straight line step sequence with +2 or +3 skill.  No one!  Only 9% of the time do they even reach +1.  A level 1 step sequence only requires three-turns, mohawks, progressives and chasses.  Steps they learned as babies, and no one can do them better than +1.

This trend follows for EVERY element, no matter how simple.  The best skaters in the world, and they are all such incompetent klutzes no one can do anything better than +1.  Or perhaps the the problem lies elsewhere.  Perhaps it is the construction of the system that leaves something to be desired.

This analysis also illustrates another problem in the judging of elements.  The following table gives all the GoEs for the 15 straight line step sequence in the Short Program

GoEs FOR STRAIGHT LINE STEP SEQUENCE LEVEL 1, LADIES SHORT PROGRAM

Fifteen different step sequences and judge 1 thought every one was the same!  Not one was better or worse than the others.  They were all just ok.  For six other judges 13 of 15 sequences were marked identical.  Only five of the 12 sets of marks move the skaters around, and then only to a limited extent.

Are the judges actually judging, or simply enjoying the best seats in the arena?  Or is there something amiss with the system?  In any event, this characteristic of the marks strains credibility.  This characteristic of the marks is also found in the other events and in other competitions.  In some Junior Grand Prix events, for example, some judges marked every step sequence with the same GoE of 0.  That is not judging.  That is seat warming.

One final problem associated with this characteristic of the marking is that it negates the expectation that something less difficult well done would earn more points than something more difficult poorly done.  In principle, the point model allows that a well done double Lutz, for example, will earn more points than a poorly done triple Lutz.  But if the judges never rate the double Lutz +2 or +3 it isn't going to happen.

Program Component Distribution.

Another areas where the marks at Europeans came up for "discussion" was the lack of spread in the marks for Program Components.  This is a well know characteristic of the marks since the system was first developed.  The following table shows the span of marks for Program Components 3, 4 and 5.

DISTRIBUTION OF RANGE OF PROGRAM COMPONENTS 3 -5

 
About 1/5 of the time the judges give the exact same mark for all three components.  Over 91% of the time, the marks had a spread of only 0.50, corresponding to a range of plus/minus 0.25 about the average mark.  The two large values (>1.00) are actually due to entry errors and not a judge's attempt to spread out their marks.

The spread in the marks for Program Components 1 and 2 is somewhat greater due to one characteristic of the Transitions mark.  This mark is frequently different from the other Program Components, but this is not actually due to any greater spreading out of the marks.

The following graph shows the difference between the marks for Skating Skills and Transitions for the ladies Short Programs.

What this graph shows is that the marks for Transitions average 0.25 less than the marks for Skating Skills, but that the spread of the Transitions marks is still just plus/minus 0.25 with respect to the average.  The marks for Transitions track with all the other program components with just an added offset (shift) of 0.25 points. Note also that the band of symbols is equally wide in the graph from left side of the graph to the right.  That means for the best skaters, the worst skaters and the middle skaters, their Skating Skills and Transitions were rated nearly equivalent in all case.

By inter-comparing all the Program Components, one finds that all five marks track together, with little spread in the marks.  The only difference between the five marks is the offset in the Transitions mark.  As we have written elsewhere, the idea that every skater performs every Program Component skill equally well is absurd, and we view this as a defect in the system.

Ordinals

Although ordinals are not calculated in the new judging system, they remain a useful tool for analysis of the judging process and the results.  Some examples follow.

There are two things to notice in the following table.  First, the order of finish based on all skills (using all judges) does not follow the official order of finish in four cases.  These are the cases where random selection of the judges has contaminated the results.  Second, the rankings for jumps, spins and sequences show considerable variety for most skaters.  The system does reasonably well in distinguishing between these skill categories, and they are clearly being judged independently.  For PC 1-2 and PC 3-5, however, the rankings show very little variation.  Each skater was ranked in these two skills categories in basically the same order.  This is another symptom of the failure to use the Program Components independently.

RANKING IN SKILL CATEGORIES FOR LADIES FREE SKATING

Next we look at the placement and points each skater received from each judge.  Notice the variety among the point totals from judge to judge, and the fact there is as much variety among the placements in the new system as there ever was in the former system.  There has been no change in that regard, and over many competitions it appears there is more spread in the new system than in the old.  This says there there is a  lot of "noise" in the system.  Also note that under the majority principle Poykio would place ahead of Liashenko.

POINT TOTALS AND PLACES FOR LADIES FREE SKATING

When broken down by skill category we find that the difference of opinion among the judges for jumping skill corresponds to about 4-6 places for each skater.  Again this is a symptom of the excessive noise in the results; something that is seen in the total points table and all the other ordinal tables below.

ORDINALS FOR JUMPS ONLY

For spins we find that the difference of opinion among the judges for this skill corresponds to about 5-8 places for each skater.

ORDINALS FOR SPINS ONLY

For sequences we find that the difference of opinion among the judges for this skill corresponds to about 5-8 places for each skater.  Note the large number of ties for any one judge.  This is symptomatic of the fact many judges give the same GoE for nearly all sequences in an event.

ORDINALS FOR SEQUENCES ONLY

For the first two program components we find that the difference of opinion among the judges also corresponds to about 5-8 places for each skater.

ORDINALS FOR PC 1-2

For the last three program components we find that the difference of opinion among the judges also corresponds to about 5-8 places for each skater, with the worst case example 15 places.

ORDINALS FOR PROGRAM COMPONENTS 3-5