Hal Varian pointed me to this article in The Economist:
Instrumental variables help to isolate causal relationships. But they can be taken too far
“Like elaborately plumed birds…we preen and strut and display our t-values.” That was Edward Leamer’s uncharitable description of his profession in 1983. Mr Leamer, an economist at the University of California in Los Angeles, was frustrated by empirical economists’ emphasis on measures of correlation over underlying questions of cause and effect, such as whether people who spend more years in school go on to earn more in later life. Hardly anyone, he wrote gloomily, “takes anyone else’s data analyses seriously”. To make his point, Mr Leamer showed how different (but apparently reasonable) choices about which variables to include in an analysis of the effect of capital punishment on murder rates could lead to the conclusion that the death penalty led to more murders, fewer murders, or had no effect at all.
In the years since, economists have focused much more explicitly on improving the analysis of cause and effect, giving rise to what Guido Imbens of Harvard University calls “the causal literature”. The techniques at the heart of this literature–in particular, the use of so-called “instrumental variables”–have yielded insights into everything from the link between abortion and crime to the economic return from education. But these methods are themselves now coming under attack.
Instrumental variables have become popular in part because they allow economists to deal with one of the main obstacles to the accurate estimation of causal effects–the impossibility of controlling for every last influence. Mr Leamer’s work on capital punishment demonstrated that the choice of controls matters hugely. Putting too many variables into a model ends up degrading the results. Worst of all, some relevant variables may simply not be observable. For example, the time someone stays in school is probably influenced by his innate scholastic ability, but this is very hard to measure. Leaving such variables out can easily lead econometricians astray. What is more, the direction of causation is not always clear. Working out whether deploying more policemen reduces crime, for example, is confused by the fact that more policemen are allocated to areas with higher crime rates.
Instrumental variables are helpful in all these situations. Often derived from a quirk in the environment or in public policy, they affect the outcome (a person’s earnings, say, to return to the original example) only through their influence on the input variable (in this case, the number of years of schooling) while at the same time being uncorrelated with what is left out (scholastic ability). The job of instrumental variables is to ensure that the omission of factors from an analysis–in this example, the impact of scholastic ability on the amount of schooling–does not end up producing inaccurate results.
In an influential early example of this sort of study, Joshua Angrist of the Massachusetts Institute of Technology (MIT) and Alan Krueger of Princeton University used America’s education laws to create an instrumental variable based on years of schooling. These laws mean that children born earlier in the year are older when they start school than those born later in the year, which means they have received less schooling by the time they reach the legal leaving-age. Since a child’s birth date is unrelated to intrinsic ability, it is a good instrument for teasing out schooling’s true effect on wages. Over time, uses of such instrumental variables have become a standard part of economists’ set of tools. Freakonomics, the 2005 bestseller by Steven Levitt and Stephen Dubner, provides a popular treatment of many of the techniques. Mr Levitt’s analysis of crime during American election cycles, when police numbers rise for reasons unconnected to crime rates, is a celebrated example of an instrumental variable.
Two recent papers–one by James Heckman of Chicago University and Sergio Urzua of Northwestern University, and another by Angus Deaton of Princeton–are sharply critical of this approach. The authors argue that the causal effects that instrumental strategies identify are uninteresting because such techniques often give answers to narrow questions. The results from the quarter-of-birth study, for example, do not say much about the returns from education for college graduates, whose choices were unlikely to have been affected by when they were legally eligible to drop out of school. According to Mr Deaton, using such instruments to estimate causal parameters is like choosing to let light “fall where it may, and then proclaim[ing] that whatever it illuminates is what we were looking for all along.”
IV leagues
This is too harsh. It is no doubt possible to use instrumental variables to estimate effects on uninteresting subgroups of the population. But the quarter-of-birth study, for example, shone light on something that was both interesting and significant. The instrumental variable in this instance allows a clear, credible estimate of the return from extra schooling for those most inclined to drop out from school early. These are precisely the people whom a policy that sought to prolong the amount of education would target. Proponents of instrumental variables also argue that accurate answers to narrower questions are more useful than unreliable answers to wider questions.
A more legitimate fear is that important questions for which no good instrumental variables can be found are getting short shrift because of economists’ obsession with solving statistical problems. Mr Deaton says that instrumental variables encourage economists to avoid “thinking about how and why things work”. Striking a balance between accuracy of result and importance of issue is tricky. If economists end up going too far in emphasising accuracy, they may succeed in taking “the con out of econometrics”, as Mr Leamer urged them to–only to leave more pressing questions on the shelf.
“Instruments of Development: Randomisation in the tropics, and the search for the elusive keys to economic development”, by Angus Deaton, NBER Working Paper No. 14690, January 2009
“Comparing IV with Structural Models: What Simple IV Can and Cannot Identify”, by James Heckman and Sergio Urzua, NBER Working Paper No. 14706, February 2009.
“Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)”, by Guido Imbens, NBER Working Paper No. 14896, April 2009
“Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments”, by Joshua Angrist and Alan Kreuger, Journal of Economic Perspectives 15(4), Fall 2001
“Let’s take the con out of econometrics”, by Edward Leamer, American Economic Review 73(1), March 1983
This news article seems excellent to me. I think it helped that Guido was probably their key source. I just have a few comments:
1. Regarding the effects of the death penalty (the first example given above), I strongly recommend this article by Donohue and Wolfers.
2. Many of my recent thoughts on causal inference, instrumental variables, and econometrics appear in this review of the recent instant classic book by Angrist and Pischke. To very briefly summarize:
– I think much of the discussion of different flavors of average treatment effects misses the point: interactions are important. (See section 3 of the review.)
– Instrumental variables and other “identification techniques” are often presented as a way of getting at some difficult causal question. But I think often it goes the other way, that a researcher starts with the natural experiment and analyzes the causal flow from there. (See section 4 of my review.)
– Here’s my personal trick for understanding instrumental variables analyses. Give it a try: it works for me!
Prof. Gelman,
I'm a little appalled by this post. You quote an entire article and do not provide a link to it. The latter is just bad form, but the former is a clear violation of copyright.
Ceolaf: What I did was copy an message that I'd received in my inbox. The email had no link in it, so I didn't really think about the issue. In any case, I just did a web search, found the article, and put in a link. So if you're interested in the article in its original form, you can just click above.
Even with the link to the article, I have to second ceolaf's general point. First, it's obviously illegal to quote the article at this length, though this doesn't _necessarily_ mean it's wrong. But it's hard to see how it could be OK. I think most bloggers, or at least most bloggers who think about the ethics of what they are doing, don't think it's right to quote all or even most of an article written by somebody else, especially from a for-profit source: if we deprive these places of their audience, they will stop producing their work. And we obviously think this work has value, or we wouldn't want to quote it! Hmm, here's a blog post by somebody who has been on both sides of this.
I encourage you, in cases like this, to quote enough of the article to serve as a teaser for it; provide a link to it; and give as many comments as you wish. If The Economist is smart, they'll cross-link to your post: "if you liked this article, you might be interested in Andrew Gelman's insightful comments…"
I agree that what you are doing is great for your readers, at least in the short run: I haven't gone to The Economist's site to read the article, nor am I going to: there's no need, I just read their article here! So it's good for me, but that doesn't mean it's fair to them.
When writing my previous comment, I checked The Economist's site to see how old the article was, and whether you could have provided a link. I typed "instrumental variable" into it's search field and got this article as the first link. It's from last week. It might have taken me 30 seconds, but was probably less than half that.
When it comes to copyright, there are legal guidlines for fair use — though no absolute formula. I think that they make sense morally, not just legally. (They are examined holistically, rather than any one of them necessarily being sufficient to sway things either way.)
1) The purpose and character of the use. In this case, it is not even academic criticism. You write that it is "excellent," and "just have a few comments," comments that are fairly minor and do not really require any qouting of the article at all. Heck, your comments are about the topic, not the article. Your purposes seem to be sharing an intesting article (bad), to remind people of other works (bad) and using it to draw attention to your own earlier work (bad).
2) Nature of the copied work. It's not a private work. OK. But it's not just a bunch of facts, the story of a recent public event or anything of that nature. It's an opinion piece, sort of. It's an explanation of something interesting. I don't think that this pushes things either way.
3) Amount and substantiality of the piece copied. This is where you lose, both legally and morally — lose so badly that none of the other factors could overwhelm it. You did not have to quote the whole thing to make your point. Heck, you admit that you didn't make any effort to find a link to it before your original post. The fact that you got it in an email message with out a link is neither a moral nor a legal excuse. You republished the whole piece! That's copyright infringement.
4) Effect on the works value. Well, it's not a huge impact, but certainly an impact. This blog is publicly accessible and shows up highly in google search results. (Try the following search: instrumental variables the economist) Some readers will find it here, and not there, denying them advertizing revenue.
*********************
Why do I care? How would you (and your publisher) feel if someone posted complete pdfs of your books on their website? That would not be legal, and it would not be moral.
Academics are knoweldge workers, and those who get paid for writing their books — or aspire to it — are the last people who should be violating copyright. I'm not suggesting that you've gotten rich from your books, but you've made some money. The fact that Red State Blue State made the money it did likely means that your next general interest book will get you even more money.
I'm in academia, too. And it's worse in my institution — a school of education — because educators are as bad about this as academics. But rarely rarely do I such blatent violations. Sure, teachers at all levels copy articles and chapters in abundance, and that's bad. But that's private and limited in its impact. It also serves a clear educational purpose. It's something we need to address throughout education and especially in academia.
Ceolaf:
I respect that this is an important topic for you. I copied the article straight from an email I received. A lot of emails come in, and I don't always google to find where the links came from. In general, when I have the link, I try to excerpt as appropriate. If you look at my general practice on this blog, I think you'll find that we're in close aggreement on blog linking and citation practices.
I disagree with you that it's bad for me to share an interesting article (but I agree that a link and excerpt is better). I disagree that it's bad for me to remind people of other works, and I disagree that it's bad to remind people of my own thoughts on the topic. That's one big reason for this blog–to share and to connect ideas.
P.S. Red State, Blue State did not make a lot of money for us. But I'm glad that we wrote the book; I think this was an effective way of getting our message agross.
Thanks for the article, I am starting my dissertation in Econ and have taken a field exam in econometrics, and taught an intro course this summer. I was shocked to see an article about IV in the Economist, I have been reading it for 30 years (I am an old guy who went back to school) but I love the Economist for its wide ranging, and in this case, rather narrow and specific, reporting. I'll look at your posts too, thanks for the links. I found it interesting that Heckman would be looking at this, he has done a bunch to keep us on the right path already, so thanks James!
I only meant bad from the persective of strengthening a fair use claim. They are good things, but they don't justify the unauthorized republishing of someone else's work – at least not from a legal perspective.
I still do not understand why you cut and pasted the entire article, though. Couldn't you have excerpted from it without the link? Given the whole article, why would you need the link to exerpt from it?
You see, the issue it not the lack of a link. I am not accusing you of plagiarism, and while your original citation obviously is not enough for academic purposes, whether or not is enough for a blog is secondary to the copyright issue.
In fact, this sort of wholesale repulication of other's work is a big issue these day. The AP is concerned about blogs doing exacty what you did. On the Media did a piece on this last month.
Obviously, bloggers want to be able to use larger/more excerpts than the original content producers want them to. And the standards of commericial media are clearly quite different from those of the academic world. But I don't see where amid all those standards reprinting someone else's whole piece is acceptable.
No interactions – No Flavors
RE: "different flavors of average treatment effects misses the point: interactions are important"
The presence of interactions is what makes different ways of averaging matter.
Agree the point likely is being missed by many but perhaps because its too obvious…
Keith
coelaf: I suspect Dr. Gelman is just accustomed to passing PDFs around of working papers and drafts. And academic publishing has never been much of a money-maker, for the publisher or for the author.
When you get into this habit of treating all sorts of information as having zero monetary value, you sort of forget how touchy the issue can be to people who do make a lot of money from copyright. That's why Dr. Gelman completely misunderstood your comment — instead of condensing the article into a paragraph or two, he instead thought that leaving the full text up was acceptable, so long as a link is present.
Putting principles aside and getting to practicalities: this blog probably doesn't come close to showing up on the Economist's radar, given that it's frequented mostly by academics with site licenses. This is the "if a tree falls in the forest" argument. It may have been wrong, but does it actually matter?
Anyway, this sort of thing is endemic on the web these days. I think it's too late to stop it. Everytime you search for anything on Google, you get "In order to show you the most relevant results, we have omitted some entries very similar to the XYZ already displayed." I've seen the full text of very expensive textbooks posted, chapter-by-chapter, on blogs.