National Bias in Judging Revisited

Following the 1997/98 season we did an analysis of national bias in judging at the Olympics and World Championships.  We found that judges from all countries placed skaters from their own country higher than the panel as a whole.  For some countires this bias was very minor, as low as one fifth place on the average for the Russian judges, while for other countries the bias was extreme - in the worst case 2.5 places on the average for the judge from Denmark. We have repeated this analysis for judging at the 1999 World Championships and provide the results here, comparing them to what we found in 1998.

The details of how we do this analysis can be found in the archive but here is a brief summary.

The difference between where a judge places a skater and the panel as a whole places the skater is called the deviation.  If a judge places a skater higher then the panel the deviation is negative, while if the judge places the skater lower the deviation is positive. If the judge and the panel agree the deviation is zero.

We first calculate the deviations for all skaters and all judges for all events.   We then calculate the average deviation for all judges from each country in all events in marking their own skaters.  In the absence of national bias the average deviation should be zero.  A positive result occurs when judges from a given country tend to rate their skaters lower than the full panels on the average, and a negative value occurs when judges rate their skaters higher than the full panels.  The results of these calculations are shown in the following table together with the results we obtained in 1998 for comparison, and the percent change from 1998 to 1999.  Based on the typical spread intrinsic in judges' marks we claim, somewhat arbitrarily, that an average deviation in the range 0 to -0.67 is reasonable performance with respect to national bias (blue in the table), -0.68 through -1.33 is poor performance (black in the table), and an average worse than -1.33 (red in the table) indicates a serious problem with national bias.

The results obtained in 1999 are similar to what was found in 1998, but several differences are also noted.  The most important conclusion from these results we claim is that, overall, national bias as measured by these statistics declined in 1999 compared to 1998 for all but two countries.  Only 38% of the countries were rated adequate for national bias in 1998 versus 62% for 1999.  Also of note is the result that judges from five countries were on-panel for their skaters and one country even placed its skaters below the rest of the panel in 1999.  Neither of these situations was seen in 1998.  The fact that the Finnish judges placed their skaters below the panel is even more remarkable considering that the 1999 Worlds was held in Helsinki.  We tentatively attribute this improvement in 1999 to a reduced temptation to succumb to national bias in a non-Olympic year in which pressure on a country's skaters to succeed is less intense.

Of the countries for which we can compare results in both 1998 and 1999 we find that 86% of the countries rated acceptable with respect to national bias in 1998 are also rated acceptable in 1999, while 71% of the countries rated poor for national bias in 1998 were rated mediocre in 1999.  In other words, the countries showing the least national bias in 1998 also tended to show the least national bias in 1999 while the countries showing the most national bias in 1998 also showed the most national bias in 1999 - though at a reduced level.  We again call attention to the judges from Canada who were among the worst for national bias in 1998 and were only surpassed by Estonia for national bias in 1999.  Further, of all the countries, Canada showed the lowest percentage improvement in 1999, trailed only marginally by Switzerland.

Return to Title Page