At the Deutsche Bank Group think tank, I have spotted Are elite universities losing their competitive edge? by Han Kim, Morse and Zingales. It's an interesting application of multilevel modeling. I won't write too much because the abstract is self-explanatory:
We study the location-specific component in research productivity of economics and finance faculty who have ever been affiliated with the top 25 universities in the last three decades. We find that there was a positive effect of being affiliated with an elite university in the 1970s; this effect weakened in the 1980s and disappeared in the 1990s. We decompose this university fixed effect and find that its decline is due to the reduced importance of physical access to productive research colleagues. We also find that salaries increased the most where the estimated externality dropped the most, consistent with the hypothesis that the de-localization of this externality makes it more difficult for universities to appropriate any rent. Our results shed some light on the potential effects of the internet revolution on knowledge-based industries.
Here is a plot of research output (measured in journal publications) given the number of post-PhD years:
My main "complaint" against the paper is that "measuring" research productivity in terms of quantity or citation impact is asking for trouble: With Goodheart's law, it's very easy to optimize for number of publications (splitting research into the smallest publishable bit), citations (cite your friends and have your friends cite you), impact of the journals you publish in (polish the paper so that it glitters, and sprinkle it with impenetrable mathematical mystique). What really matters is stuff that people will read and be affected by it. Most of the good papers I have read in the past few years weren't published in an elite journal: I have read drafts, circulations, web pages. And most of the time I spent reading elite journals was a waste of time.
Aleks,
I don't know how relevant 1 data point is, but when I looked up my own 20 most-cited articles, all but one came out between 1990 and 1998. The one exception was my 1995 article on prior distributions for multilevel models, which finally appeared in Bayesian Analysis (not a "top-tier journal") but whose citations have basically all come from its life as a preprint.
On the other hand, if I have a preprint that never gets published, I'm afraid it might just disappear from memory. So, based on my experiences (and congruent with your comments above), I tend to prefer publishing somewhere and to minimize the hassles of any particular journal.
I've been told that things are different in economics, however.
I am not sure how valid your criticism of the productivity measures are if you are not willing to sugest an alternative (placement of PhD students in other top tier department is an alternative, do you liek it better?) Furthermore in economics (particularly in theory, less so in other areas such as development) some journals have virtual monopolies on the truly earth shattering papers in their subfields (econometrica in micro theory is the foremost example, the QJE in the last few years in empirical work appears to be headed in the same direction).
Econgeek, the first step towards finding a solution is formulating the problem.
Aleks,
thanks for this post! The way "we" measure performance in academia is really not very much academic. For those of us who are not "top-shots", there is always the question of whether we optimze what is measured, or whether we "just" do interesting research. Unfortunately, this is often not the same!
For now, I can't come up with a better way of evaluation either, but as Aleks said:"The first step is formulating the problem".