From Jessica, I saw a review by “Econjeff” of my review of Joshua Angrist and Jorn-Steffen Pischke’s new book, “Mostly Harmless Econometrics: An Empiricist’s Companion.”
Econjeff pretty much agrees with what I wrote, but with one comment:
I [Econjeff] am a bit surprised by Gelman’s call for more on hierarchical models; I think economists are right to treat these as a combination of useful pedagogical tool for education research design and an unnecessarily functional-form dependent way to get the standard errors right when then the unit of treatment differs from the units available in the data.
I think this is a common perception of multilevel (hierarchical) models among economists. Regular readers of this blog will not be surprised to hear that I disagree completely! The purpose of a multilevel model is not to “get the standard errors right” but rather to model structure in the data.
An analogy that might help here for economists is time series analysis. If you have data with time series structure and you ignore it, you can get over-optimistic standard errors. But that’s not the main reason people do time series modeling. The main reason is that the time series structure is interesting and important in its own right. We are interested in individual and contextual effects and unexplained variation at the individual and group levels, just as we are interested in autocorrelation, periodicity, long-range dependence, and so forth.
See chapters 1 and 11 of ARM for more discussion of motivations for multilevel modeling.
i think that there's a much more fundamental reason that many economists don't see a pressing need to care about multi-level models. unlike statisticians, they often only care about aggregate behavior and look for effects only in the population.
for example, it has long been thought that mixed logit models solve the independence from irrelevant alternative property (iia). while it's true that the mixed logit does not possess iia at the population level of the model, it is also true that the mixed logit does not solve the real underlying issues of the model. it's another example of the ecological fallacy.
i'll link to the paper that explains why if it catches your interest.
tom
p.s. i really enjoy your blog.
Tom,
Not to disagree with you but to say it more generally: I think economists typically think that their goal, when doing a statistical analysis, is to estimate some prespecified "beta." From that perspective, there are only three things that can be done:
(a) Reduce bias;
(b) Reduce the standard error;
(c) Get a better estimate of the standard error.
(There's no such thing as "get a better estimate of the bias, because this would immediately be inserted into the estimate as a bias correction.)
Multilevel models don't necessarily do (a), and they typically do the opposite of (b), so the natural assumption is that their main benefit must be (c).
From a statistician's perspective–and maybe from the perspective of non-econometrically-trained social scientists in general–the purpose of a statistical analysis is not to estimate some predetermined beta but rather to understand a system. Better modeling can lead to better understanding.
Again, I think the time series analogy could be a good one for economists, since there I think it's more common for models to be used to understand a system rather than simply be used as a way to account for annoying data difficulties that get in the way of estimating beta.
tom, could you post the link to that article?
thanks
the purpose of a statistical analysis is not to estimate some predetermined beta…
the purpose of a multilevel-modeling analysis, right? Otherwise this comment seems (uncharacteristically) dismissive
Daniel: Tom's article is at the link to his name by the comment.
Anon: I'm not trying to be dismissive. What I meant was that, to a typical economist, the purpose of an analysis is to estimate beta. To a typical statistician, the purpose of an analysis is to model data or to make a prediction. All these perspectives have merit.
to a typical economist, the purpose of an analysis is to estimate beta
I would add that this is true of some economists like Angrist and Pischke but certainly not all economists.
As someone who teaches a core econometrics course to PhD students, the main reason we don't use multilevel models is because we don't know what they are. I'd have to learn them on my own (I have learned just a little bit by reading your books) and then add it to the lectures without being able to offer any empirical examples.
Add that to the fact that it is difficult enough to get referees to read papers using methods they understand, let alone using methods they don't understand, and there is little incentive to move in that direction.
Perhaps if you would write a paper showing how and why economists should use them, and give some simple examples, the methodology would be used by economists.
I'm not sure economists are interested in statistical developments outside their own discipline…very parochial area…
Wow … someone read my blog!
I am more sympathetic to multi-level models than most economists (not a high bar, but one that depends a bit on how you count those who have never encountered them), and have been known to defend them on occasion, but I am remain unsure about what they add relative to a linear model with both low level and high level variables and standard errors that take account of clustering and such. That is the reason for my focus on their value at clarifying the underlying model – what I call their pedagogical value, though this is likely not the best term – rather than their statistical virtues.
One response to my concern would have to do with random effects, and I am very sympathetic to the emphasis in the MLM literature that variables and treatments may not have the same effect on all units. But this, too, can be accomplished without the multiple levels and, to some degree, without the maximum likelihood apparatus and associated functional form assumptions.
Oh, I should note too that reading Andrew's review caused me to buy his (co-authored) book. Now if someone would provide me with the time to give it a careful reading, I would be all set!
Andrew: I hope we cross paths in person at some point. I know many who sing your praises.