Sean Schubert pointed me to this article, ” Figure Skating Scoring Found to Leave Too Much to Chance”:
The overseers of international figure skating scoring instituted a new system in 2004, designed to reduce the chances of vote fixing or undue bias after the scandal during the Winter Olympics in Salt Lake City in 2002. Under the old rules eight known national judges scored a program up to six points with the highest and lowest scores dropped. Under the new rules, 12 anonymous judges score a program on a 10-point scale. A computer then randomly selects nine of the 12 judges to contribute to the final score. The highest and lowest individual scores in each of the five judging categories are then dropped and the remaining scores averaged and totaled to produce the final result.
This random elimination of three judges results in 220 possible combinations of nine-judge panels, explains John Emerson, a statistician at Yale University. And according to his analysis of results from the shorts program at the Ladies’ 2006 European Figure Skating Championships, the computer’s choice of random judges can have a tremendous–and hardly fair–impact on the skaters’ rankings. “Only 50 of the 220 possible panels would have resulted in the same ranking of the skaters following the short program,” Emerson writes in a statement announcing his findings.
I have to say, selecting judgments at random seems pretty wacky to me. Why not just average all of them? I also think it’s funny that the ratings are from 0 to 10, with increments of 0.25. Why not just score them from 0 to 40 with increments of 1, or 0 to 4 with increments of 0.1?
The article does point out an interesting problem, which is that judges are perhaps giving too-low scores at the beginning to leave room later. The system of adding or deducting points from the base value seems like a step toward fixing this. But they should set the base value low (e.g., at 3, rather than at 7), so that they’ll have more resolution at the upper end of the scale. As things stand, the scale might be better designed for picking the worst skater than the best!
On the other hand, I’m not so strongly moved by Emerson’s argument that removing different judges would change the outcome. As Tom Louis has written, rankings are pretty random anyway, and, in any case, things would be different if a different set of 12 judges were selected.
It's interesting that this didn't have to be instituted for ski jumping. Eric Zitzewitz writes more.
"I also think it's funny that the ratings are from 0 to 10, with increments of 0.25. Why not just score them from 0 to 40 with increments of 1, or 0 to 4 with increments of 0.1?"
The right answer is : why not ?
Can human being really notice 40 distinct degrees of quality?
Gladwell's "Blink" mentioned that 7 levels of distinction are about all that most people can handle.
Why do we accept more degrees of precision in total scores then you do in the individual sub scores?
Why not just call any score within 5 points a tie and have a skate off?
As I have Master's degrees (as well as Bachelor's degrees) in both Philosophy and Statistics, meaning I know very little about both subjects, I'm fascinated when the two subjects come together. So, in a philosophy blog (http://bengal-ng.missouri.edu/~kvanvigj/certain_doubts/) I came across this and thought it would be interesting if you could address it on your site (especially the last part):
"So a Bayesian models imprecision by a set of precise probability measures of uncertainty. Sensitivity analysis is a common method of using imprecise models to check the robustness of a (Bayesian) statistical model. (Nevermind for the moment that the reasons the Bayesian gives to justify the axioms of probability are based on precision, so are inconsistent with sensitivity analysis.)"
I know almost nothing about figure skating, but when I read that they were randomly selecting judges, I figured it was to reduce the risk bribery or other methods of coercion for influencing the outcome by outsiders. For example, would you be as likely to attempt to bribe a judge if you knew there was a 25% chance that their score wouldn't even get counted? Also, if no one knows which 9 judges are going to get selected, it would make it very hard for a group of judges to collaborate to fix the outcome.
In the article it says the random selection was "designed to reduce the chances of vote fixing or undue bias." Perhaps they introduced more randomness into the results, but with the possible benefit of reducing bias? Seems like a reasonable trade-off.
Yes, scores seem pretty "random" anyway. Can't argue with that, particularly with my background (and lack of, in figure skating)! But I don't see why that is a justification of the system of choosing 9 of 12 scores at random. Why add to the uncertainty?
More recent (e.g. Olympic) information is up on my page, now. Cheers!