Since introducing its proposed new scoring system to the public last year, the ISU has offered a variety of reasons and justifications for its development, and described various benefits it claims to offers skating. These reasons and explanations have changed over the course of time and vary to suite the situation and audience of the moment. This pattern of behavior, and the fact that all of the reasons they have offered thus far don't hold water, give the impression the ISU doesn't really know why it should be doing this.
Discussed below are the major claims the ISU has made for its proposed scoring system and why they lend little support to the need for the proposed system.
This is rarely true. The average judge probably identifies tricks correctly better than 97% of the time, and championship level judges better than 99% of the time. The misidentification of tricks rarely impacts the placement of a skater as determined by the entire panel. Assuming one trick alone would make the difference in a placement, there is about one chance in 100,000 that a misidentified trick will alter a placement. If there are judges out there who can't identify the elements, the simplest solution is to get rid of those judges through the use of a strong accountability system.
After being taught to identify the elements, the next thing judges are taught is their relative difficulty. For jumps and lifts the difficulty of the various elements are explicitly and quantitatively specified by the ISU. For the rest of the elements, it is learned at judges schools and by trial judging. If there are judges out there who don't know the relative difficulty of the elements, the simplest solution is to get rid of those judges through the use of a strong accountability system.
Just the opposite is true. It is well know, and there are hundreds of years of data and experience to show, that human judgement is much better at making comparisons than in making absolute judgements. In general, even well trained humans can make absolute judgements with only 5-10% accuracy and consistency at best. For comparative judgements, however, 1-2% accuracy, or better, is achievable. That is why skating went to a comparative system of judging so many years ago. If the top management at the ISU knew more about figure skating, or had a reasonable knowledge of the history of the sport, they would know that. One fundamental problem with the proposed ISU system is that it does not take into account the limitations of humans to judge on an absolute scale, and they have not considered the way that limitation will affect the accuracy of the results produced by the system.
Yes there are. They are called 6.0s, how many you get, and what mark you get them for.
Penalties play only a minor role in the scoring system and not in the way the ISU says when they offer this justification.
The ISU offers the following argument. When a skater first steps onto the ice they have a 6.0 and then every time they make an error they lose points. The new system will be better, they claim, because every time a skater accomplishes something in their program they will earn points. Thus, the proposed system is based on reward and not punishment.
Nonsense! This shows how the speedskaters just don't get it.
Marking in the current scoring system is based on midpoints and point ranges for each competitive level. These midpoints and ranges are based on well established program content and performance skill level criteria determined over the years. At the junior level, for example, the midpoint is 4.0. A skater who executes a program whose technical content is what is expected for a 4.0 and is performed at the skill level expected for a 4.0 will get those marks. This would be the program one would expect to see from the average junior skater in a group that spans the entire skill level of all junior skaters. If a skater executes a more difficult program, skates it at a higher skill level, or gives a better presentation than average, they will get higher marks. If a program has lesser content, or a lesser presentation it will receive lower marks. The skater is rewarded for the content of the program and its presentation at each competition level on a well established point scale. No judge is ever counting down from a 6.0 to determine a skater's mark.
In free skating and free dance the deductions are only for violations of the rules. This is no different than in other sports where you may have penalty kicks, or free throws, or touchdowns called back, due to infractions of the rules. Further, at some point deductions will have to be introduced into the proposed system to insure compliance with the rules, so nothing is going to change in this regard.
In the short programs, CDs and OD there also are deductions for specific errors. These deductions are primarily a numerical way of accounting for lesser accomplishment in the elements, and in that sense simply amounts to skaters earning lower marks for accomplishing less in their program.
So far as penalties are concerned, the proposed ISU system is actually more severe than the current system. Every element under the new system could end up with the deduction of 1 to 3 points due to errors. In principle, a skater could lose as many as 1/4 of their points under the new system due to penalties. The ISU is basically taking the deductions from the short program and now introducing them into the long program under its proposed system -- a more negative approach than is currently the case.
The new system it is claimed, will allow skaters to say, "when I started competing I was a 60 point skater, as a junior I was a 90 point skater, and now I am a 120 point skater" or "my goal is to be a 150 point skater" or some such thing.
Nonsense! That already exists.
A skater can say as a juvenile they were a 3.0 or 3.5 skater, as a novice a 3.6 or 4.2 point skater and as senior a 5.1 or 5.5 skater or whatever. There is already a well defined progression of numeric scores that skaters use to gauge their progress, and skaters already know at any given level whether their marks are good or bad for that level. If as a novice, for example, if you get marks of 3.5 you are just average, marks of 4.1 you are hot stuff, and marks of 2.7 you bite.
Yet again, the speedskaters just don't get it.
First, judges already do this -- at least according to the judges schools I have been to. Second, why just jumps? Why not difficult, complex footwork done at blinding speed, perhaps on one foot, or a multi-position blur spin while the skaters legs feel like rubber? There is more to skating than just jumps. Third, why is the ISU pushing the skaters to perform difficult, dangerous elements when they are exhausted and the risk of injury is substantially greater? This is not a race. The goal is not to show how exhausted the skaters can get. The goal is to demonstrate the best skating skills.
When hearing ISU officials discuss this one, one gets the impression they are responding to a complaint from the IOC. If so, then it is clear that ISU management is not doing a very good job of defending the sport at the IOC. Or perhaps senior ISU management can't defend the sport, because they do not understand the sport themselves.
The current scoring system is as rigorously quantitative as one can hope for using human judges, and is no less quantitative than the proposed ISU system will be. There is a fairly well established relation between program content and marks given at all levels of the sport, and for the most part judges use that relation in a consistent way. Although there is room for improvement, any limitations present are the result of the inherent limitations of human judgement. These limitations will be just as present in the proposed judging system as the current system.
Although there is some subjectivity in the current scoring system, there is not as much as people give voice to. At U.S. Nationals recently, one fan told me how a particular skater was marked down because the judges "don't like" her body type. I, on the other hand, thought she got marked down because the quality of her elements was mediocre and she made a lot of mistakes.
Part of this problem is because the ISU and the USFSA (and I assume all the national federations as well) have done a dreadful job of educating the public on what is rewarded and not rewarded in a skating program. Further, they allow TV to actively misinform the public on this very subject.
There is very little subjectivity in the first mark, limited to how seriously one views a given error. Is that a major cheat or a minor cheat? Is that a small change of edge or a major change of edge? Of the seven criteria judged in the second mark only two of these might be considered purely subjective. But more to the point here, the proposed ISU scoring system will not change this situation. There are five subjective marks in the proposed scoring system and each of the elements has a subjective quality factor as well. About 2/3 of the points in the proposed scoring system will be subjective, so the judges will continue to have plenty of wiggle room in assigning their marks.
The current scoring system has a maximum numerical score of 6.0 in each of the two marks. Over the years, as the technical content of programs has increased, the meaning of the 6.0 in the first mark has changed in most events. Consequently, one cannot directly compare a given score for technical merit today with the same score given in the distant past. The proposed ISU system is only slightly different in that respect.
Under the proposed ISU system the three presentation scores are subjective and have maximum possible point values. The two subjective technical merit marks also have maximum possible point values. For the individual elements, the four spins and two footwork sequences have maximum possible point values regardless of how difficult they may be. For jumps, the skaters will earn points for a maximum of twelve jumps. The twelve most difficult jumps that can be humanly executed defines the maximum point value that can be earned for jumps. Consequently, the proposed ISU system also has a maximum possible point limit for both the technical and presentation marks-- no different from the current situation. All the proposed system will do is redefine the point values to new maximum values, to the unnecessary confusion of skaters, judges and fans alike.
So what? Comprehensive statistics could be provided under the current system if the ISU bothered to provide them. I have been witness to many media request to the ISU since 1994 to provide more comprehensive information on the judges marks and assessments. These request have consistently gone ignored. All of a sudden this is now a reason to completely overturn the sport of figure skating? Further, the information to be provided, while of some interest, is not the information the public really wants; that being, which judges scored the competition, and what marks did they give.
The fallacy of this has been discussed at great length elsewhere and need not be repeated here in detail. In summary, however, statistical analysis shows that by using a random selection of judges, a significant fraction of the time (25% or more) the placements of the sub-set of judges will be contrary to the placements of the entire panel of judges, and as a result, the award of medals will be determined by the flip of a coin a significant fraction of the time. This prediction, which was published in June of last year, was verified by results from the Grand Prix competitions this season.
The idea that secrecy will enhance honesty is laughably contrary to human nature. If a judge does not have the integrity and character to judge honestly in the light of day they cannot be trusted to operate honestly in secret. The only way dishonesty has been uncovered in the past is through open, public scrutiny of the judges' behavior. All that secrecy accomplishes is that it allows the ISU to sweep misconduct under the rug and avoid the kind of public embarrassment it was subjected to in Salt Lake City. Result from the Grand Prix this season also indicate judges have already started to give inflated and/or biased marks knowing their misdeeds cannot be attributed to them directly due to the cloak of secrecy the ISU has given them using the interim judging system. This will only get worse as more judges catch on to the fact they can get away with murder under the interim and proposed systems.
Nonsense! Two thirds or more of the points the judges will award remain subjective and the points for the individual tricks and subjective factors are mainly additive. If you want to give a skater a helping hand, bump up each assessment one step where you want. If you want to push a skater down, drop each assessment one step. No complex math is required, and once the ISU sets the final point model, there will be only a few simple rules of thumb a judge will need to remember to know how one step in the assessment scale will affect the placement of a skater. It will be child's play and the judges will be able to do it in total secrecy.
Under the ISU Random Seven approach (picking seven random judges from a panel of fourteen) the ISU claims one would have to get to 11 of the 14 judges to skew the result, and says this is impossible. That is only true, however, in the worst case when all 14 judges would agree that a skater should be in a certain place and you wanted to force the skater into a different place.
But that amount of unanimity rarely exist in skating. Most places, in most event, in most competitions are determined with split panels. If the result is close, you need only sway one judge to the dark side of the force. It took only one judge in the Olympic pairs event to wreak havoc. Any judge who wishes to engage in national bias can skew the results in favor of their skater. In fact, in its current form, one determined judge can move any skater one place in the standings if they want to regardless of what the rest of the panel does.
It has been suggested that as the judges enter their assessments the running point total be displayed in the arena so the audience can see the point total build up during the performance. This would be analogous to the time clock in a timed event.
This shows once again the speedskaters just don't get it. This is figure skating, not a race!
While this bit of technology might prove interesting in a limited sense, it has two problems. First, the audience will/should be looking at the skating and not the scoreboard. TV, however, could use it with split screens or by showing the split point total as they do in races, but I expect most fans would just rather watch the skating. Imagine trying to watch a movie only to be periodically interrupted by an announcement "It is now 22 minutes into the movie and you have already seen 36% of the entertainment!" The last thing we need added to the nonstop chatter of TV commentators is an endless bombardment of statistics while the skaters are performing.
More importantly, however, this "innovation" would be totally misleading and would create nothing but PR problems and confusion. The running point total can only be calculated for the element points that the judges enter during the performance. All the subjective points are added at the end and these point make up the majority of the point total. In addition, judges will be able to change their assessments at the end of the performance for each trick. Consequently, the running total is meaningless. Skaters ahead of another in the running point total during the performance will end up falling behind once the final numbers are calculated much of the time. This will only confuse and annoy the public, as the ISU and TV commentators try to explain why it happened.
This practice would also give the judges an easy opportunity to misbehave. If the judges see the running point total in the arena as the skaters perform they could use that information to help or hinder a skater by bumping up their marks or reducing them for the rest of the program to manipulate the answer. Since the judges will be able to change their assessments at the end of each performance, they will again be able to use that information to help or hinder a skater and manipulate the results.
Don't like the answer after the skater is done? Go back and change your marks for all the tricks to change the result. Hardly fair.
It is difficult to believe that the IOC would actually remove the economic engine from the Winter Olympics limousine. But even assuming that is true, so far as anyone knows the IOC has not mandated any particular solution to the ISU's problems by any particular date. In fact, I was told that directly by an ISU Council member. Instead, the ISU, without adequate study or careful thought, glibly promised the ISU a specific solution last February and now is unwilling to reconsider its options for fear of looking bad. It appears the ISU is more concerned with its political position within the IOC than doing what is best for the skaters and the sport.
Yes the ISU must do something, but changing the judging system should be last thing on the list of priorities, not the first and only priority.
The ISU can, and should, take the following actions. All of these could already have been put into effect since the Winter Games. Had they done so, this would have put the ISU in the position of presenting the IOC with concrete reform within a year of Salt Lake, and not the illusion of reform three or four years later. One would think that politically, this would have been the preferable course of action as well.
At this point, one might conclude there is no case for the use of computer technology in the judging of figure skating; but that is not true. There is one reason -- and only one as far as I can see -- to use computer technology in some form of figure skating scoring system.
And while, the devil in me is thinking, "ISU, go figure it out for yourself," here is the answer anyway. Note, however, this is only a general justification to consider the use of technology, not a justification for the ISU's approach to using that technology which is flawed beyond any remote value to skating. Nor does it guarantee one can turn the available technology into a viable scoring system, due to the inherent limitations on the ability of humans to make absolute judgements accurately, as discussed above.
The value of using computer technology in a figure skating scoring system is this. Judges can be trained to identify all the elements of skating fairly easily. Judges can be trained to assess the quality of each element fairly easily. Judges can also be taught the relative difficulty of a group of elements fairy easily. Combining all of the assessments of all the elements and aspects of a skating performance to produce a final score and placement (the point model), however, is not easy. And getting a group of judges to all use the same point model is more difficult still.
Looking at ordinal sheets from many events (and statistical analysis shows) it is generally fairly easy to place the top skaters in an event and the bottom skaters in an event -- especially at the junior and senior level. The middle half of an event is another story. The top skaters usually do more or less the same tricks with similar skill levels and only a small number of errors. That makes it easy to compare and place them. The bottom skaters attempt more or less the same tricks with similar (dismal) skill levels with numerous errors each. That makes it easy to compare and place them also.
In the middle, however, one must compare skaters with jumps but poor spins, spins but poor jumps, tricks but the inability to skate, the ability to skate but poor tricks, technical skills with poor presentation skills, presentation skills with poor technical skills, etc.. Because it is impossible to train judges to use the same mental point model (not that they actually are adding up points in their heads, but a point model is in there nonetheless), the spread of placements in the messy middle is always significantly greater than at the top or bottom. From my personal experience with USFSA judges schools and in trial judging, efforts to develop a uniform point model in the minds of the judges are minimal at best, and I expect it is no different within the ISU. Much of this problem could be eliminated by better training of the judges, but no amount of training will ever completely eliminate it.
What computer technology buys you in a scoring system, then, is the uniform application of the same point model to the assessments from each judge. By taking this approach, the spread in the placements will be solely determined by the spread in the quality assessments of the judges and not the spread in the individual point models the judges are using. This eliminates one of the major "noise" sources in the results and leads to a more accurate and certain order of finish -- mainly in the middle half of the group of skaters. It is of less value for the top and bottom skaters, where the placements are more clear cut, though not always.
This approach would have even greater utility in lower level events where few skaters have reached the point they are equally skilled in all aspects of a skating performance, and so the judges end up having to compare apples and oranges for an entire group of skaters. It is not uncommon at the lower levels to have events where the winning skater has a first and a last place ordinal, and the last place skater has a first and a last place ordinal. You even see events where each judge has a different skater first. I recall judging one event with five judges and four skaters where each skater got at least one ordinal of each of the four possible places! A uniformly applied point model would eliminate most of this chaos. Unfortunately, the technology required is so expensive it is unlikely it will ever be affordable except for the highest level of competition. The system being developed by the ISU is likely to cost as much as $1,000,000 and will require a well trained technical staff to operate and maintain it. Because of that, the ISU is going to create the situation where there will be two different forms of competitive skating; one judged according to whatever point model they come up with producing results one can never trust, and one judged by the point model used by a panel of human judges.
Copyright 2003 by George S. Rossano