In a previous issue the mysterious method of marks, ordinals, and majority principle used to score skating competitions was described. This method of scoring currently finds itself under continuous attack from spectators, TV, and even some officials within the competitive skating world. Spectators tend to criticize the system due to the widespread belief that the method allows the judges too much freedom to manipulate the results. TV criticizes the system because ... well, because they are TV; and some skating officials are willing to consider radically changing the system because they are tired of hearing the judges jeered by spectators and are sensitive to the perception that the system is corrupt.
It is frequently suggested that skating should adopt a system in which the high and low marks are dropped and the total points of the remaining marks used to determine the placements. At the October 1995 meeting of the USFSA Board of Directors a proposal to adopt this system was discussed. Further, the USFSA ProAm competitions have used this approach since the fall of 1995. In the following we will refer to the current system as the "ordinal" method, and the suggested replacement system the "clipped total" method.
When setting up a scoring system for a sport like ice skating which, due to the complexity of the factors that are considered, must be judged by a panel of people, several factors must be taken into account. First and foremost, no matter how hard you try you will never find a panel of people with equal knowledge, observational skills and priorities, and who are capable of judging with perfect consistency on identical absolute scales. Not gonna happen, not ever. The reality is, no matter how well trained, people will always posses different levels of knowledge, different observational skills, and place different emphasis on the many factors taken into account in a competition. Judges award their marks using slightly different numerical scales which are subject to small random inconsistencies. They also can make more serious mistakes; sometimes whoppers. Finally, human nature being what it is, the possibility of marks with a more nefarious purpose must be protected against.
The ordinal system was set up taking the above factors into consideration. It is not the original, or even second, method of scoring used over the past 100 years of modern competitive ice skating. Previous methods have included using a ten point scoring scale, final placements based on total points, and carrying point totals between the different parts of an event. The ordinal method was developed to avoid (or at least minimize) the weaknesses and abuses of former systems.
Human judgment is not particularly good at quantifying observed phenomena on an absolute scale with great accuracy or consistency. Even with care, human judgment is rarely accurate to better than 5% in an absolute sense, corresponding, for example, to 0.3 points on skating's 6.0 scale. Human judgment, however, is extremely good in making comparative decisions. Unlike the clipped total method, the ordinal method is specifically designed to take this into account.
Following the first skater's performance, the judges are given the median mark for that skaters marks. This provides the judges a consistent starting point for comparing the subsequent performances. In addition, the judges know that in events of different level the marks will typically lie in an accepted range. This guarantees that the judges' marks will be in generally the same ballpark for all the skaters of an event, but does not guarantee identical marking scales for all the judges. The ordinal method, however, does not require identical marking scales since all skaters after the first are marked on a comparative basis.
The marks the judges give are only a means to determine the ordinals. While marking an event the judges keep track of both the marks they have given and the order of finish they have specified. In fact, judges decide on the placement first and then assign marks consistent with that placement. Judges try to give marks that reflect their view of the performance on an absolute scale, but this is only approximate, and if a judge must give a "weird" mark to place a skater in the order they think appropriate, then a weird mark it is. From a mathematical perspective, the use of ordinals is far superior to the clipped total method to determine the order of finish when a group of judges are marking a competition using different numerical scales.
If judges marks were subject only to small random errors, any number of methods could be used to combine the ordinals to determine the order of finish. Marks, however, are subject to unintentional goofs and personal biases (both unintentional and not). The majority principle is designed to minimize the effect of errors and biases in determining the order of finish. If the majority of a panel decides that a skater should be placed no higher than 5th, for example, no matter how high one judge places that skater he still ends up 5th. Similarly, if one judge places that skater 30th he still ends up 5th.
Although not immediately obvious, the majority principle does in fact "ignore" judges' marks that are way out of line with the rest of the panel, and does it better than the clipped total method. Only when a panel is nearly equally split can the ordinal from one judge change a skater's order of finish, and then it can do so typically by only one place. In some situations one judge's marks can change a skater's order of finish by two places, but this is relatively rare and requires a panel in serious disagreement with each other. In comparison, statistical analysis shows that even after throwing out the high and the low marks, the clipped total method is more adversely affected by errors or biases of a single judge than the ordinal method, and is extremely poor when two or more judges have similar errors or biases. The clipped total method always throws away the two most extreme marks whether they are in error or not, and at the same time always includes intermediate marks that are off-panel. The ordinal method, on the other hand, always ignores all ordinals from the panel that are out of line and only those that are out of line.
Consider, for example, results where first and second place are determined on a five-four split. If five judges have skater "A" first, the opinions of the four other judges are irrelevant and skater "B" is second. There is nothing the minority judges can do to change the result. Compare that to the clipped total method. In the clipped total method the high and low judges' marks are thrown out and skater "A" would probably end up with four first place ordinals and three second place ordinals, still a majority of firsts. However, since we would use the total points to determine the finish, if the three remaining judges give skater "B" high enough points, skater "B" will win even though a majority of the judges place skater "B" second. Further, if the judges are marking on distinctly different scales it is conceivable both the high and low marks are first place marks for skater "A", in which case the skater with a minority of firsts from the whole panel would win without any hanky-panky on the part of the minority judges.
In a five-four split the total points for the two skaters involved typically differ by only a few tenths, and often by only one tenth. In the clipped total method, one judge has only to skew the marks by two tenths - 0.1 in each mark, for example - to overrule the majority opinion in many cases. For two judges with similar perspectives it is easy to overrule the majority opinion most of the time. Overall, there is much less protection from the effects of errors and biases in the clipped total method than in the ordinal method. In addition, while errors and biases are fairly easy to identify in the ordinal method, they are less obvious in the clipped total method. If a judge places a skater too high or low by several places it is abundantly obvious in the ordinal method. In terms of the point totals used in the clipped total method, however, this error in placement might correspond to only a few tenths of a point, far less obvious. Thus, not only is the clipped total method easier to manipulate, it is also easier to hide what it going on.
| Ordinal Method | Clipped Total Method |
| Hard to understand | Easy to understand |
| Hard to use | Easy to use |
| Based on the comparative nature of human judgment | Ignores the comparative nature of human judgment |
| Tolerates differences in judges' marking scales | Intolerant of differences in judges' marking scales |
| Results highly immune to off-panel marks by a single judge | Results only sometimes immune to off-panel marks by a single judge |
| Minimizes the effect of errors or biases on results by multiple judges in all cases | Minimizes the effect of errors or biases by one or two judges in only some cases |
| Highly immune to manipulation of marks | Somewhat immune to manipulation of marks |
| Hard to hide gross errors and biases | Easy to hide gross errors and biases |
The main problem with the ordinal method is that it is hard to understand. Couple that with the failure of the ISU, NGBs and TV to educate the public on how it works, it is not surprising to find the public suspicious of the method. What is surprising, however, is to find that rather than aggressively moving to educate the public, some within the competitive skating world instead seek to avoid public criticism by converting to a scoring system that is clearly inferior to the one currently in place.