Statistical vs. practical significance

Shravan asks,

Suppose I do a multilevel model analysis of repeated measures reaction time data and some predictor that I have for reaction time shows a significant HPD interval/p-value. But when I look at the estimate I find that the effect is about 1 millisecond. This means in effect that that particular effect only contributes a one ms slowdown to reaction time. But it's significant statistically; sometimes it is hugely significant.
If one is just looking for whether the effect has an impact on reaction time, not the magnitude of the effect, should one declare evidence in favor of the effect? Or should one say, well, it *is* statistically significant, but what does a slowdown of 1 ms mean? Our hardware only samples events only 4 ms. Under this second view, I would be forced to say that in practice this is not an important effect.

A more general question related to the above came up in a review of a paper of mine recently. In our paper we had shown that a large effect that had been found previously in the literature becomes tiny (but remains significant) when we control word frequency. The reviewer objected that this finding may be important for computational modelers, who make quantitative predictions about that effect, what why should experimentalists care? I.e., why should people who only make qualitative (boolean) predictions about the effect care about the result, when the outcome is qualitatively the same?

My own understanding is that practical significance matters even in such theoretical explorations, and whether one implements a computational model or is reasoning qualitatively about data is irrelevant.

My reply:

Peter

Lots of subtle points here, which means they require thought. Saying 'significant at 0.05' doesn't require thought (although the analysis itself may require lots of thought. So....why would anyone be willing (even eager) to spend money, time, and thought on a study and then leave the decision about its importance to a process that forbids thought?

Jor

this kinda thing happens in bio-medicine alot with biomarker studies. Where some marker is significantly correlated with outcomes -- but when you use it for classification of patients into different treatment groups, and construct ROC curves, you don't really get much better performance than using just standard clinical data.

Statistical Modeling, Causal Inference, and Social Science: Statistical vs. practical significance

Statistical vs. practical significance

Peter

Jor