Ladies Results Men's Results Pairs Results Dance Results
The effect of an obvious and extreme entry error on the results (as occurred yesterday) received a great deal of discussion today in the press room. And while to the media I heard ponder the issue it seemed obvious this was a problem that needed a solution (and for which a simple solution exists) the response from the ISU was under-whelming.
The reaction of two ISU representatives to this problem was the usual ISU knee-jerk defense and spin of the proposed system. The initial reaction: it's not a concern; it's the judges responsibility to enter the marks correctly and one can't constantly be holding their hands; this type of error would have even a greater impact in the old system; it has no impact on the results. Eventually it was conceded that it could have a small effect in the point totals, but not enough to worry about.
Here is the really check. In the ordinal system and in OBO this type of error would rarely, if ever affect, the results. In the proposed system, an entry error of the type that occurred yesterday can introduce an error of over 1 point in the point totals -- and one didn't have to wait long to see the possible impact of this. Later in the day, second and third place in the Dance event were separated by less than one-tenth of a point, and in the Pairs event first and second place were separated by a little over one-tenth of a point. An entry error for any of these four teams would have altered the medal results in these events.
This situation also sadly illustrates the complacent, superficial approach the ISU is taking to development of this system, and the frustrating unwillingness of the ISU to concede the existence of even the simplest of problem and to readily embrace a simple solution, or to even consider the suggestion of a solution.
The close results in the Dance and Pairs events also illustrates another point that has been previously discussed elsewhere on this site. The system must have a very high degree of statistical accuracy for the results to be meaningful. In its current form, a difference of one-tenth of a point is statistically meaningless and so are the corresponding placements. Skaters in this situation are in effect tied, and at worst incorrectly placed.
The results in the Pairs event also again illustrates the problems inherent in randomly selecting judges. For the judges actually chosen, Pang & Tong placed second in the free skate and first overall in the Pairs event. However, a different random choice of judges in the free skate would give Petova & Tikhanov two additional points and a bump up to first place. So who really deserved to win the gold medal in Pairs? Depending on the random choice of judges you get two different answers.
Yesterday two of the weaknesses of the ISU point model were illustrated in the Men's event, when the winner of the short program was determined by the credit given for a failed element, and the marks for one skater had to be recalculated after the fact due to a defect in the ISU scoring software. Today, four more examples showed up in the day's events.
In the Ladies short program one judge mistakenly entered a mark of 0.75 for one of Sasha Cohen's five subjective marks and did not notice it until it was too late to correct the error. Input errors cannot be corrected from the instant the skaters' marks first appear on the arena scoreboard. Although such an error will have only a small impact on the results, it does have an effect, and it could result in the change of a place or the loss of a medal. It is also a completely unnecessary error that illustrates the careless approach the ISU is taking in the development of the system. A key part of system reliability for a system that depends on multiple data entries from multiple users in multiple steps, is error checking for the inputs to insure data integrity. This is something (error checking) that should have been built into the software design from day one, not something to think about adding a year into development, should they even choose to do so. In this particular case, the software should be checking the inputs for potential errors and flagging the judge to verify the data input before the results are posted.
In the Dance original dance, simple inspection of the judges marks, not available in the arena, shows that the winner of the OD was determined by the random selection of judges. Seven of the eleven judges awarded Belbin & Agosto higher marks than Delobel & Schoenfelder, but they ended up second in the dance. No doubt there are other examples to be found in a careful analysis of the marks, but this case is the most obvious.
In the OD one also finds an example of the failure of this system to control bias, the main purpose for its existence. One of the judges, the French judge we assume, had the French team scored 10 points (15%) higher than the other ten judges. Because the judges' marks are not displayed in the arena, this obvious bias was hidden from the skaters and public view. Because of the secrecy surrounding the handling of bias and accountability we will never know who this judges is; and more importantly, what "corrective action" might be taken - if any - to deal with this situation.
In the Men's free skate the illogic and danger inherent in the new approach to counting cheated quads as triples was illustrated. Weiss had planned a quad toe loop - triple toe loop, a quad toe loop, and a triple Axel - quad toe loop. On the opening combination he did a triple - triple. He two footed and perhaps slightly under-rotated the quad, which was designated a third triple toe loop. The successful triple toe loop after the triple Axel, was then his forth toe loop.
What this says is, because a flawed quad toe loop will be considered a triple, if you have two quad toe loops in a program together with two triple toe loops you must be very careful because errors in the quads may result in two of the four jumps not counting; whereas in the past, the jumps would be accepted for what they really are, two failed quads and two triples, not four triples. In the proposed system, a skater can lose many points depending on the order in which the jumps are executed. For example, if the quads come first and are designated triples, the subsequent triple toes will not count and the skater might lose up to 15 points.
Fans in North America got their first direct look at the proposed ISU scoring system here at Skate America. About 3000 spectators witnessed use of the new system in the Austrian Waltz, and the Men's and Pairs short programs. The audience was very subdued with the most energetic displays reserved for Michael Weiss and the other American skaters. For much of the time, however, the audience sat in quiet confusion trying to understand the new system and the marks displayed in the arena.
The reading of the marks has always been a lengthy affair that has made skating competitions drag on interminably. Because of the more than passing interest in how the judges awarded their individual marks, in the past this process at least kept the audiences' attention. Beginning last season, however, with the hiding of the judges' individual marks, the process has become just plain boring and of little interest to the fans.
At this competition each event began with a very lengthy but confusing description of the point system. After each skater, a series of eight numbers were read aloud and then the current place of the skater. Other than the total, what these numbers meant appeared to be of little interest to the audience, which for the most part seemed restless for the next skater to get on the ice.
With this new system, and its automatic nearly instantaneous calculation of points, all that is needed is a display of the numbers and an announcement of the current total and place. It should take about 15 seconds. Instead, in the Compulsory Dance over an hour was spent to watch 20 minutes of skating. For the men's and pairs short programs about an hour and one-half was spent in each event to witness less than 30 minutes of skating. Think of watching an hour TV program with 40 minutes of commercials.
In the men's short program Michael Weiss won by virtue of generous credit for a two-footed, under-rotated quad toe loop attempt, while Ryan Jahnke had his point total recalculated overnight due to yet another bug in the poorly and inadequately tested ISU software.