Pseudo-failures to replicate

Prakash Gorroochurn from our biostat dept wrote this paper discussing the fact that, even if a study find statistical significance, its replication might not be statistically significant–even if the underlying effect is real.

This is an important point, which can also be understood using the usual rule of thumb that to have 80% power for 95% significance, your true effect size needs to be 2.8 se’s from zero. Thus, if you have a result that’s barely statistically significant (2 se’s from zero), it’s likely that the true effect is less than 2.8, and so you shouldn’t be so sure you’ll see a statistically significant replication. As Kahneman and Tversky found, however, our intuitions lead us to (wrongly) expect replication of statistical significance.

Prakash’s paper is also related to our point about the difference between significance and non-significance.

1 thought on “Pseudo-failures to replicate

  1. A similar idea can be found in
    @ARTICLE{Killeen2005,
    author = {Killeen,Peter R.},
    title = {An Alternative to Null-Hypothesis Significance Tests},
    journal = {Psychological Science},
    year = {2005},
    volume = {16},
    pages = {345-353},
    number = {5},
    doi = {10.1111/j.0956-7976.2005.01538.x},
    eprint = {http://www.blackwell-synergy.com/doi/pdf/10.1111/j.0956-7976.2005.01538.x},
    url = {http://www.blackwell-synergy.com/doi/abs/10.1111/j.0956-7976.2005.01538.x}
    }

Comments are closed.