Greg Mankiw links to this article by Peter Lawrence on the mismeasurement of science. Lawrence writes:

Modern science, particularly biomedicine, is being damaged by attempts to measure the quantity and quality of research. Scientists are ranked according to these measures, a ranking that impacts on funding of grants, competition for posts and promotion. The measures seemed, at first rather harmless, but, like cuckoos in a nest, they have grown into monsters that threaten science itself. . . .

The journals are evaluated according to impact factors, and scientists and departments assessed according to the impact factors of the journals they publish in. Consequently, over the last twenty years a scientist’s primary aim has been downgraded from doing science to producing papers and contriving to get them into the “best” journals they can. Now there is a new trend: the idea is to rank scientists by the numbers of citations their papers receive. Consequently, I predict that citation-fishing and citation-bartering will become major pursuits. . . .

I have a few scattered thoughts on this. First, it is kind of funny to rate a paper based on the citation counts of the journal where it appears. But I guess that might make sense for a fresh new paper that hasn’t had a chance to get cited. (An exception would be this paper, which got cited a lot before it ever was published, but that was an unusual case due to unforeseen delays with a journal that was just starting up.)

Second, I think it makes sense to separate two concerns: (a) criticisms of citation indexes for evaluating existing research, and (b) concerns about incentives that will distort future research. I expect that there is a high correlation between the quality of scientists’ research programs and their citation counts. But I could see problems heading into the future.

Another issue is citation inflation. At least if you check on Google Scholar, you’ll find that citations have increased lots just in the past few months–I assume they’ve either added new journals to the database or gotten better at linking citations to articles.

You also have to be careful comparing researchers in different fields. Biology and computer science get lots more citations than statistics.

Finally, I don’t quite understand Mankiw’s response. He links to Lawrence’s article, which would seem to indicate some agreement with it, but then he also links to a list which puts him at #6 among all economists. These views aren’t necessarily inconsistent–it’s possible for a measurement system to be crappy but to still give reasonable results in individual cases–but it’s not clear to me what Mankiw’s actual views are. In particular, as an economist he might have some view of the importance of these rules as incentives. (The list itself is interesting to look at. I’ve heard of only a few people on the list, so I can’t be sure, but it looks like you have to go down to #66 to find a woman. (Not that things would be much different in a list of statisticians.) I have to say, though, the idea of such a list is pretty unpleasant to me. I mean, how would it feel to be, say, #124 on the list? Would it be frustrating to be so low, or just cool to be on the list at all?