John Cook discusses the John Tukey quote, “The test of a good procedure is how well it works, not how well it is understood.” Cook writes:
At some level, it’s hard to argue against this. Statistical procedures operate on empirical data, so it makes sense that the procedures themselves be evaluated empirically.
But I [Cook] question whether we really know that a statistical procedure works well if it isn’t well understood. Specifically, I’m skeptical of complex statistical methods whose only credentials are a handful of simulations. “We don’t have any theoretical results, buy hey, it works well in practice. Just look at the simulations.”
Every method works well on the scenarios its author publishes, almost by definition. If the method didn’t handle a scenario well, the author would publish a different scenario.
I agree with Cook but would give a slightly different emphasis. I’d say that a lot of methods can work when they are done well. See the second meta-principle listed in my discussion of Efron from last year. The short story is: lots of methods can work well if you’re Tukey. That doesn’t necessarily mean they’re good methods. What it means is that you’re Tukey. I also think statisticians are overly impressed by the appreciation of their scientific collaborators. Just cos a Nobel-winning biologist or physicist or whatever thinks your method is great, it doesn’t mean your method is in itself great. If Brad Efron or Don Rubin had come through the door bringing their methods, Mister Nobel Prize would probably have loved them too.
Second, and back to the original quote above, Tukey was notorious for developing methods that were based on theoretical models and then rubbing out the traces of the theory and presenting the methods alone. For example, the hanging rootogram makes some sense–if you think of counts as following Poisson distributions. This predilection of Tukey’s makes a certain philosophical sense (see my argument a few months ago) but I still find it a bit irritating to hide one’s traces even for the best of reasons.
Completely agree. In biostat there is an awful tendency to display a new statistic or method without saying what on earth the thing is doing. If there's an implicit model that you show me, then I can think about what it may or may not be robust to without trying all possible simulations. In the same vein, understanding what the method is getting at allows one to modify or extend it. The down-side of that is your biological collaborators suggesting all the faults of that model (even if they are faults which your method is robust to).
From a practical standpoint, the bigger concern is not whether the method is understood (by statisticians) but whether the method can be explained to the clients. Lots of complicated methods are almost impossible to explain to non-specialists. The risk is that the clients are likely to trust the method just because they produce results they like, and conversely to reject the method when the results disappoint.
I'm reading this paper on education using an econometric style method invoking IV. The authors did an audacious job of trying to fish out a subpopulation and retroactively define "randomized" test and control groups. It took them some 10 pages to make the argument that they have found a randomized experiment. If it takes 10 pages to explain why comparison groups were randomly selected, it's probably not randomly selected.