Ranks have lots of problems. They’re statistically unstable (see the work of Tom Louis) and can mask nonlinearity. I was recently reminded of these patterns in seeing two sets of graphs reproduced by Kaiser:
From the Wall Street Journal, graphs of baby names over time:
The graphs are ok but plotting ranks, rather than proportion of total names each year, is a mistake, since it makes the y-axis extremely hard to interpret, it’s not clear where zero is, etc. As Kaiser points out, in any case there are difficulties when the scales are different for different plots, but, beyond this, the ranks are making things tougher.
And, from the New York Times, a summary of problems with the subway lines:
Here, the ranks are giving a hyperprecision that is not helpful. (Also, encasing the subway line numbers/letters in black circles makes them harder to read, at least on the screen.) As some commenters pointed out, it would probably be better to just display each line with three or five grades, sort of like how Consumer Reports does the ratings.
Andrew,
Could you be a little more specific about that? Ranking works just fine when you are trying to model consumer choices (e.g. paired comparisons), which are non-metric to begin with.
In the name graphs, zero is infinitely far down, no? If I remember correctly names roughly follow a power law, i. e. that the frequency of the nth most common first name is proportional to n^(-α) for some constant α.
This particular transformation has the effect of enlarging small fluctuations and shrinking large fluctuations; my guess is that Nicole, for example, would seem to be losing its popularity much faster if plotted in a more conventional way.
It's possible that only rank data was available, though.
Bill,
I have very little experience modeling consumer choices, but Bradley-Terry and similar models allow one to estimate continuous distributions of preferences using paired comparison data. But really my complaint is with treating the ranks as a goal in themselves. I also dislike some of Tukey's paired comparisons stuff from the 1950s where he tried to estimate rankings of treatments. In general, I can't think of many examples where we should particularly care about rankings.
Isabel,
I'd rather have zero at zero. One option, as discussed by Kaiser, is to have the plots on two scales: a common scale for all the names and another scaled to the maximum for each name to see details.
Also, I think non-rank data are available, since they're at the baby name site.
Andrew,
Thanks. I agree, the ranks themselves are not the goal; however sometimes they are the only observable that makes sense.
I was thinking about the problems with ranking in surveys of consumer choices yesterday. Think about the task of putting 10 products in order, from worst to best. The difference between the top two items might be much smaller than that between the bottom two, or the other way around.
You might be able to tease this relationship out with a lot of ranking data, but a better way to get this information might be to place the items in roughly proportional order from best to worst using a 1-D slider… This way, you get the ease of ranking along with the precision missing from just putting the items in order.
What do you think?
Professor Gelman – would you happen to have a particular paper by Tom Louis in mind that you could point me to?
I know that in my lab we use the "tied rank" transformation a lot in order to tease out signal from the global background noise, but I've always wondered what the theorists have to say about it.