Test failures

Jimmy brings up the saying that the chi-squared test is nothing more than “a test of sample size” and asks:

Would you mind elaborating or giving an example? Hypothesis tests are dependent on sample size. but is the chi-squared test more so than other tests?

And setting aside the general problems of hypothesis testing, off the top of your head, what other tests would you consider useless or counterproductive? (For new and infrequent readers, Fisher’s exact test.)

My reply:

I like chi-squared tests, in their place. See chapter 2 of ARM for an example. Or my 1996 paper with Meng and Stern for some more in-depth discussion.

To answer your later question, I think that most “named” tests are pointless: Wilcoxon, McNemar, Fisher, etc. etc. These procedures might all have their place, but I think much harm is done by people taking their statistical problems and putting them into these restricted, conventional frameworks. In contrast, methods such as regression and Anova (not to mention elaborations such as multilevel models and glm) are much more open-ended and allow the user to incorporate more data and more subject-matter information into his or her analysis.

4 thoughts on “Test failures

  1. See Berkson (1938) in JASA for the first time, at least that I know of, that this was brought up in the literature. You can get it on JSTOR here.

  2. I agree with the general point. The best example is the "Fisher's exact test". If the word exact is not in the name, I bet it is used less.
    But I want to bring up a practical point: a test is much easier to explain to a non-statistical audience than regressions, multilevel models, etc. Explaining how models work is very hard when you have interaction terms and many variables and random effects, etc. It's like saying "trust me". So it's a tradeoff between better fit and ease of understanding.
    I usually do both. The model results should agree with the test results. And often I can present the test findings but I know that they are supported by modeling too.

Comments are closed.