Judea Pearl sends along this article and writes:
This research note was triggered by a discussant on your blog, who called my attention to Wooldridge’s paper, in response to my provocative question: “Has anyone seen a proof that adjusting for intemediary would introduce bias?”
It led to some interesting observations which I am now glad to share with your bloggers.
As Pearl writes, it is standard advice to adjust for all pre-treatment variables in an experiment or observational study–and I think everyone would agree on the point–but exactly how to “adjust” is not always clear. For example, you wouldn’t want to just throw an instrumental variable as another regression predictor. And this then leads to tricky questions of what to do with variables that are sort-of instruments and sort-of covariates. I don’t really know what to do in such situations, and maybe Pearl and Woolridge are pointing toward a useful way forward.
I’m curious what Rosenbaum and Rubin (both of whom are cited in Pearl’s article) have to say about this paper. And, of course, WWJD.
I think Pearl risks perpetuating the debate by continuing to say things like "examples abound showing that certain covariates may actually increase bias if included in the analysis." The problem is that "included in the analysis" is not clearly defined by Pearl. Perhaps it would be better to say "included in the types of analysis researchers often employ." As you (Gelman) have mentioned previously, he seems to imply that "included in the analysis" means including these variables on the right hand side of a regression. I agree that this is often how analysis is done, but it misrepresent Rubin's point: to a Bayesian there is no scientific basis for excluding variables from analysis, broadly defined. Critically, "including in the analysis" does not imply that variables are included on the right hand side of a regression. It simply means that a Bayesian will always condition on all the observed data, including intermediate outcomes, colliders, and what not. As far as I can tell, there is no scientific principal to justify not conditioning, in the Bayesian sense, on all observed data.
It is possible that under some assumptions the posterior distribution of the object of interest will not require us to model a collider or intermediate variable. This would be similar to the arguments Rubin makes again and again about the special role of ignorable missing data mechanisms. However, as far as I can tell, Pearl has simply shown that if you include intermediate variables in the wrong way, they can create problems. Has he shown somewhere that these variables provide no information? This seems doubtful, because intermediate variables typically provide lots of information about final treatment effects.
Dear Andrew,
Thanks for posting my note, and for pointing out
that adjusting for an IV can be problematic, if done indiscriminately.
Your note, however, seems to imply that throwing an IV as another regression predictor is such an obvious mistake that it would not cross the mind of the prudent.
I must confess that I did not know about the IV
danger before reading Wooldridge's paper —
IV is not an intermediate variable, so the danger is not shown in the graph.
I am convinced Rubin did not suspect it either when he wrote (in a discussion on propensity score methods):
"To avoid conditioning on some observed
covariates… is distinctly frequentist and non-scientific ad hockery." But my question is about how to detect such dangers in the future, in the
narrow context of propensity score methods for causal effect estimation.
Indeed, following the best advice in the mainstream literature,an IV looks just like any other confounder:
0. It is a pre-treatment variable.
1. It is related to treatment
2. It is related to outcome
3. It is related to outcome conditioned on treatment.
4. It is not an intermediate variable.
Thus, why would anyone suspect this innocent looking bystander of such nasty behavior
as amplifying bias.
I am curious to know if you can recommend an intuition that would alert researchers to the dangers of using an IV as a regressor in a predictor.
Dear Oliver,
Your points are well taken.
I am about to revise my paper with your suggestions:
1. Replace:
"example abound showing that certain covariates
may actually increase bias if included in the analysis."
with:
"example abound showing that certain covariates
may actually increase bias if used in propensity
score matching, or in an inverse probability weight for causal effect estimation"
2. Likewise, I will take special precautions to make sure that Bayesians do not view my paper as advocating the exclusion of data from correct analysis.
Finally, I am curious to know how arguments about the role of ignorable missing data mechanisms can protect us from using an IV in the wrong way, namely, in the context of propensity score matching.
The pdf file linked appears to have a problem. Pages 2 and 3 was in blank, at least in my pc. Is there a problem with the file or is my pc?
Best Regards,
Manoel