The Senior Men's event at the 2007 U.S. Nationals resulted in a tie score and the champion was crowned based on a tie breaker. Now what are the chances of that happening?
Pretty damn minute.
But not as small as one would think. In a small number of results at non-qualifying competitions we checked from this season, there were a few ties in individual event segments and in total scores. Roughly speaking, ties show up every few hundred starts when there are five or fewer judges on the panel. They are less common as the size of the panel increases.
Beyond ties, results determined by a few hundredths of a point are more common. The inquisitive can scan results since IJS was introduced and you will find several medals at ISU events awarded based on point differences of a few hundredths of a point, and many lower places determined by that slim margin.
Extrapolating our small sample to an entire competition season in the U.S., it can be expected there could be several hundred ties, and several thousand places determined by a few hundredths of a point. Most of these cases will not be medal places, but some, obviously, will.
Given that background, hopefully it is obvious that if results and medals are going to be decided by point differences less than one-tenth of a point -- or even zero -- then the calculation method must be precise to the nearest 0.01 point. There is no point in using scores calculated to the nearest 0.01 point if the calculation is not precise to 0.01 point.
The question to be addressed then, is whether the current calculation method for IJS is actually precise to 0.01 point, and if it isn't how bad is it.
To answer the question, one first needs to decide what is the TRUTH to which other calculation methods will be compared.
Hopefully the choice of Truth method will be obvious. It is the calculation method carried out with full mathematical precision without any rounding. If the final sum is then rounded to two decimal places, then a score reported in that way will depart from the truth by no more that .005 points.
In the IJS calculation method, however, some of the intermediate results are rounded to two decimal places, resulting in an accumulation of small errors compared to the exact calculation. What we want to know is how large an error can accumulate by doing a sloppy calculation with intermediate rounding.
To quantify this we have taken all the results from all event segments for a competition with a large number of starts this season and calculated scores by doing an exact calculation, and two different calculations with intermediate rounding. We then compare the three sets of calculated scores. The large number of starts involved make this a statistically significant sample.
In the following, we will refer to the three sets of scores as:
Truth -- the exact calculation with no rounding
Method A -- the GoE trimmed mean is calculated and rounded to two decimal places for each element, the Program Component (PC) trimmed mean is calculated, multiplied by the required weighting factor for the event, and then rounded to two decimal places.
Method B -- the GoE trimmed mean is calculated and rounded to two decimal places for each element, the Program Component (PC) trimmed mean is calculated and rounded to two decimal places, multiplied by the required weighting factor for the event, and then rounded to two decimal places.
The difference in approach for Methods A and B are noted in bold.
In the following illustration we show the difference in score between the Truth calculation and the Method A calculation. Note that rounding by Method A introduces an error of up to 0.03 points high or low. This means that should one skater round high and the next placing skater round low, an error in the score of 0.06 points can be introduced by rounding.
Next we illustrate the difference in score between the Truth calculation and the Method B calculation. With this method, the departure from the truth is even greater. In this case errors of 0.05 are introduced, which can result in point difference errors of up to 0.10 points.
For both of the above examples notice that with rounding virtually all the results are affected to some extent by carrying out a sloppy rounded calculation. Of the two methods, Method B is clearly the worst of the two, with point errors 67% greater with Method B compared to Method A.
We illustrate the difference between using Method A or Method B in the following graph. The difference in calculated score for the two rounding methods is up to 0.05 points for one skater. This illustration clearly shows that averaging, factoring and rounding (Method A), does not produce the same results as averaging, rounding, factoring and rounding (Method B) for over 50 percent of the starts in this sample of starts.
The above calculations show what the typical effects of rounding are for a large number of starts. The theoretical worst case is actually much worse.
By rounding each term in the point sum separately, an error of up to 0.005 is potentially introduced. If by (admittedly rare) luck every term in the sum rounded the same way by the maximum amount, an error of 0.095 could be introduced in an event with 14 elements and 5 PCs. If one skater rounded up 0.095 and another down 0.095, then an error in the point difference for the two skaters could reach 0.19 in principle due to sloppy calculations. The chance of this happening, of course, is minute. Nevertheless, our real-world examples shows that errors half the potential maximum show up several times in a competition with several hundred starts. (Note, in some non-qual competitions there may be as many as 1200 starts or more.)
The above calculations are for a single event segment. In an event with multiple event segments the errors from the individual segments can either combine in the same sense and increase the total event error, or cancel out to some extent and reduce the total event error. With two event segments, such as in singles and pairs, the errors will combine to increase the total error half the time and the total event error can be up to twice the error discussed above. In dance, with three event segments, the total error can triple compared to a single event segment, one-quarter of the time.
Given that sequential places with a point difference of 0.10 or less regularly occur a few percent of the time, it is absurd to use a calculation method that introduces arithmetic errors of that amount, and potentially more. It is imperative, therefore, in the interest of fair competition that the calculation method used in IJS be changed at the 2008 ISU Congress to eliminate intermediate rounding, to provide the athletes a point total that is precisely calculated.
If the calculation method is changed the chance of ties will be significantly reduced, but it can never be eliminated. A related item, then, that has our undies in a wad, is what to do about skaters who end up tied in total points. The current process is to break the tie using the point totals from the Free Skate or Free Dance.
This process is a carry over from the Total Factored Place method of determining results under the 6.0 system. In that method, this approach had merit. The Free Skate was clearly much more important than the Short Program. Skaters who won the Short Program but not the Free Skate rarely won the event. But this approach is an anachronism in a point based system.
Under IJS, the Short Program and the Free Skate are equally important in determining final results. A skater who wins the Short Program but not the Free Skate wins the event half the time. Why then break the tie based on the Free Skate? Just because it was skated last? In football it would be like saying if the score is tied after 60 minutes, the team with the most points in the second half wins. Or in baseball, the score from the last six inning breaks the tie if the game is tied after nine.
In no other sport is such a silly tie breaker used. In most every other sport, if the game is tied at the end of regulation play, there is overtime, or a shoot out, or something to make a decision. In hockey you can actually get tied games. In races, if the times are the same the athletes are tied.
The same should now be true for skating. If the skaters are tied after the Free Skate or Free Dance they should be tied. Something equivalent to overtime play, or a shoot out would probably be more exciting for the fans, but would probably be impractical, and deciding what it should be probably unobtainable.
So that leaves a tie. If two skaters end an event with the same score, that is supposed to mean they showed equivalent skill sets. In that case the right thing to do is give them same medals and not make the decision based on an archaic method of tie breaking that is not consistent with a point based scoring system.
Round in the current IJS calculation method introduces unnecessary errors into calculated point totals and competition results. The use of intermediate rounding in the calculation method should be removed at the next ISU Congress
The use of tie breakers in determining results is an anachronism in a point based scoring system. The use of tie breakers should be removed at the next ISU Congress. Skaters with equal total points at the end of an event should be considered tied.
Copyright 2008 by George S. Rossano