Dan Kahan writes:
Hi. I’m wondering if you — and readers of your blog — have a take on how to preempt the mistake of construing overlapping confidence intervals as indicating that distinct predictors (e.g, two treatments in an experiment) do not have a significantly different effect. See Schenker, N. & Gentleman, J.F. On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals. Am. Stat. 55, 182-186 (2001). The mistake is common enough to make me fret about using really nice bar plots w/ CIs when the CIs overlap. One can always point out in the text that it is a mistake to see overlapping CIs as indicating lack of a significant difference, and then report the relevant difference of the relevant point estimates & the CI associated with *that* difference, but since having to use additional text to explain how to interpret a figure undermines the whole point of using a figure, I’m wondering if there are better reporting or graphic-display strategies I’m unaware of.
My reply: I’m not as worried as you might expect by this, as statistical significance is pretty arbitrary anyway. I’m more worried about people not realizing that the difference between “significant” and “not significant” is not itself statistically significant. Ultimately, if there’s a particular comparison you want people to make, you have to make it yourself, and if there are any comparisons that you don’t want people to make, it’s best to explicitly tell them not to do it.
Two good discussions of this topics, with different "solutions", are:
1) Cumming, G. Inference by eye: reading the overlap of independent confidence intervals. Statistics in Medicine, 2009, 28:205-220.
2) Goldstein, H., Healy, M. The graphical presentation of a collection of means. JRSS-A, 159:175-177.
You could use beanplots. That would make the size of the overlap readily apparent. http://cran.r-project.org/web/packages/beanplot/i…
Given two point estimates and their associated confidence intervals, say est1 (lower1, upper1) and est2 (lower2, upper2), it is easy to obtain a confidence interval for the difference. As Zou and Donner (2008, Stat Med 27: 1693-1702) and Zou (2008, Am J Epidemiol 168: 212-224) show, the answer is
Lower = est1 – est2 – sq root of ( square of(est1-lower1)+ square of(upper2-est2) )
Upper = est1 – est2 – sq root of ( square of(upper1- est1)+ square of(est2-lower2) )
In fact, these formulae work for a wide range of problems. In addition to provide a direct answer to the question of 'how big the difference is', these results promote interval estimation, rather than pull thinking back to hypothesis testing, as done by Cumming's (2005, 2009) rule of eyes.