Tested for only a few competitions in only one season, we now are up to version 3 of CoP. Or is it 30 or 300? There have been so many changes over the past eight months it is hard to keep track.
First released last summer for limited use this season, the system was changed on almost a weekly basis and still remains a moving target. Proposals for the ISU Congress were supposed to be submitted by December 1st, but CoP was still in such disarray, only a pseudo, shell of a proposal was submitted.
It wasn't until February that a revised proposal based on experience this past season was completed and posted on the ISU website. That version lasted less than two months, and on April 15 the "final" proposal for the Congress was posted. The buzz has it, however, the ISU is already working on changes for another version, and if there is a way, legal or otherwise, to slip more changes into the agenda between now and June, don't be surprised to see version four discussed at the Congress.
By the counting of the ISU, 50 changes were made to CoP after the Grand Prix Final in December 2003, and since February additional changes gave been made. So here it is less than two months before the Congress and the ISU development team is still making significant changes to the system. Could it be any clearer? This is still a make it up they go along rush to get something approved, for an experimental system that is just not ready!
The approach ISU management is now pushing is "pass it now, fix it later." Unfortunately, once it is passed, the ISU members lose all bargaining power to correct the problems that exist in the system. The only leverage Members now have to get changes in the system is to withhold their approval until the changes they feel necessary are made.
In addition, ISU management is proposing that the Council be given the right to change the rules whenever they want, without the approval of the Members through a Congress. In addition to being able to change the point values through a Communication (which is probably a good thing), ISU management is asking for the right to change the rules, the skating standards, the fundamental structural details of the scoring system, everything, all at their own discretion, as frequently as they wish without accountability to anyone.
Would any other sport allow a small group of individuals more concerned with politics and personal power than with the well being of the sport the authority to change the sport on a weekly basis without accountability and without adequate testing of the consequences of their actions? Would football teams and their fans support a system where the rules for each game were different from one weekend to the next, set by a cloistered body that was unaccountable to anyone? The power ISU management is asking for to control the rules is too great for any sports governing body, and the history of the ISU shows that ISU management cannot be trusted with such power.
Given the problems that still exist in CoP, it is in the best interest of skating and the ISU Members that the Members adopt the approach "fix it now, pass it when its ready," where ready means fully tested, and not likely to be subjected to frequent significant changes.
CoP has not reached that point. The system must now be considered untested in its current form even for small events (in terms of number of competitors), such as the Senior Grand Prix, due to the large number of frequent changes that have been made since the Grand Prix Final. The system is completely untested for Championships. The system is completely untested for Junior Grand Prix events. The system is completely untested for events with 15-30 competitors. The system is completely untested for lower quality Junior events. The system is completely untested for Novice events and below. The manual form of the system is completely untested at all levels. The system is completely untested for use in domestic club competitions. Given this state of affairs, pass CoP in haste, repent at leisure. The Russian federation has referred to CoP as an experimental system they do not want used at the 2005 Moscow World Championships. They are correct in their characterization that CoP is still experimental, and are right to be skeptical about its use in their event.
And what of the 50 or more changes that have been made? Is CoP any better? Have any of the fundamental problems been corrected?
For the last question, the answer, sadly, is no.
Most of the changes that have been made have minor impact on the system, one way or the other, though overall they represent a small improvement. Among the few significant changes, some have been for the better but more have been for the worse. In addition, none of the fundamental structural and mathematical problems with the system have been corrected.
For the better, cheated quads will no longer be counted towards the restrictions on the repetition of triple jumps. These jumps will, however, still receive substantial credit, even though the attempt at the quad is in reality a complete failure. Also for the good, the classification approach for lifts in free dance has been substantially improved.
The "Plushenko problem" has been "solved" by allowing a third jump combination in singles free skating. This "solution", however, entirely misses the point and is completely ineffective for the real problem. The "Plushenko problem" is the fact that the system is so complicated it is too, too easy for a competitor to make a minor improvisation in their program only to find their score decrease, resulting in a loss of a place, or (as Plushenko found out the hard way) a gold medal. If an experienced competitor such as Plushenko can fall afoul of the complexity of this system what chance does an inexperienced competitor have?
A further consequence of this change has been to tilt the balance of free skating even more towards jumps. Before this change, jumps could make up about 42% of the score in men's free skating, and presentation about 30%. By adding yet another jump to the programs (and increasing the values of the triple and quad jumps), this increases to about 45% for the jumps and about 28% for presentation. The idea that CoP selects the best well rounded skater is absurd. Free skating might be viewed as a 4 1/2 minute pentathlon consisting of jumps, spins, step sequences, fundamental skating skills and presentation. In CoP, however, just one of the five skills makes up nearly one-half of the score. A skater who is superior to another in four of the five skills is at an disadvantage if the skater superior in jumps builds up a lead in jumps alone of as little as 4-6 points, and is at an extreme disadvantage for a lead in jumps of 8 points or more (basically one jump of the 12 allowed in the men's singles).
Another area that received attention since the Grand Prix Final was the point values for jump combinations. It was recognized by the ISU that the point values assigned to combinations was often inconsistent with the actual relative difficulty of the combinations. This brilliant flash of insight is indeed accurate, and the problem is due to the simple minded way CoP scores jump combinations and sequences. It is a fundamental structural problem of the algorithm (mathematical process) used. Rather than solve the fundamental problem with the algorithm, however, the ISU changed the point values for the triple and quadruple jumps. The purpose of this was mainly to reduce the value of triple toe - triple toe and increase the value of triple Axel - double toe. To accomplish this, the value of triple toe loop and some of the other triples were reduced and the value of the quads increased. This, however, has several undesirable consequences.
First, the point values of triples and quads has now increased overall, which (together with adding the extra jump) unbalances the already unbalanced mix of credit for jumps, spins, step sequences, etc.
Second, this further unbalances the already unbalanced mix of credit for singles, doubles, triples and quads. By tuning the points for triples and quads in senior level events, the balance of elements is thrown out of whack for all other levels of skating (i.e.; for junior and below), and the recent change in the triple and quad point values compounds this problem.
If one tries to apply CoP to all competitive levels in domestic U.S. competitions (as ultimately will be the case if CoP passes) one finds that it is impossible to get a reasonable balance between, jumps, spins, step sequences, etc. that is consistent among all levels. Only by making significant structural changes to CoP is it possible to vaguely approach a reasonable balance of elements, and even then the balance among the elements varies wildly for each competition level. In creating CoP, the ISU development team apparently gave no thought to whether CoP produced reasonable and consistent results at all competition levels. As it turns out they developed a system that is fundamentally incapable of delivering that; and because the problem is in the fundamental structural characteristics of the system, no amount of fiddling with the point values will make that problem go away.
Penalties for falls have been increased. Now, if a skater falls in crossovers, or standing there doing nothing, there will be a one point deduction. In dance, the deduction will be one point if one partner falls and two if both fall. In the past, penalties for falling have primarily been limited to falls in short program elements. Now all falls will cost the skaters. It is ironic that a system that was supposedly introduced in part because the current system is too punitive, is turning out to be the most punitive system ever used by the ISU.
The guidelines for judging the five subjective program components have been extensively revised. The judges struggled with these marks during the Grand Prix, and even cursory examination of the marks show that the judges were incapable of using the five marks independently. There is no evidence that the new guidelines are any better, or that the judges will be able to use the five marks independently in the future. Without testing it is just a wild guess whether the new guidelines will solve the problem of the judging of the program components.
On the subject of anonymity and random selection of judges, the ISU has made a small compromise. In all the minor events, anonymity and random selection will not be used. In the events that matter such as the Grand Prixs, Championships and the Olympics the system will continue to be wrapped in secrecy, with a sub-set of the panel randomly selected (and consequently the medal winners randomly selected a significant fraction of the time).
Basically, the ISU is throwing a bone in the hope it will distract the skating community from the evils of anonymity and randomness that will continue to permeate the most important events. In this regard, it is interesting to note that the IOC requires that all scoring systems used in the Olympic games be "transparent". Nevertheless, the ISU continues to push forward a system that is neither transparent nor accountable to the public and the skaters.
At the recent World Championships while discussing anonymity, ISU president Cinquanta stated unequivocally that under no circumstances will ISU management tolerate a scoring system which allows the public or the skaters the opportunity to question the marks from the judges. With no ambiguity he confirmed that a goal of the new scoring system is to hide the marks of the judges from the skaters and public so that they can never again be questioned. In his own words, he in effect confirmed that the goal is to hide problems not to solve them!
Another change to the system is a reduction in the size of the panels, and the use of only a single trimmed mean instead of a double trimmed mean. The number of judges marks that actually go into calculating the results will remain at five.
From a mathematical point of view, use of a trimmed mean to filter bias from the calculations is a poor substitute for more sophisticated methods, but if a trimmed mean is to be used, then a double trimmed mean is vastly superior to a single trimmed mean. For Championships, the Russian federation is even proposing the use of a triple trimmed mean!
The use of only five marks to calculate the scores is a fundamental structural defect in the system which leads to results that have no statistical accuracy and thus no validity in close competition. The use of a single trimmed mean provides little protection against biased marks or other systematic errors of judgment. Rather than decreasing the size of the panels they should have been increased and the double trimmed mean retained. By eliminating the undesirable random selection of judges, one could even have a smaller panel (than the 14 used in the interim system) and a double trimmed mean with seven marks going into the calculation. But that is not what is being proposed by the ISU.
Changes to CoP since the Grand Prix Final include a number of small steps forward and a few large steps backwards. The fundamental structural defects in the system remain, and are not even on the table for discussion at the ISU. CoP in its current form is not ready for use. Adopting CoP at this time is neither advisable nor necessary. A more prudent course of action would be to eliminate the atrocity that is the interim system with the Modern Era 6.0 system that is on the Congress agenda, and continue development of CoP at a small number of events next season. The Modern Era 6.0 system effectively addresses all of the major problems identified after the Salt Lake Olympics and meets all the concerns of the IOC regarding the scoring of figure skating.
Copyright 2004 by George S. Rossano