Kenneth Burman writes:
Some modern, computer intensive, data analysis methods may look to the non-statistician (or even our selves) like the equivalent of the notorious Rube Goldberg device for accomplishing a intrinsically simple task. Whereas some variants of the bootstrap or cross validation might fit this situation, mostly the risk this humiliation is to be found in MCMC-based Bayesian methods. I [Bruman] am not at all against such methods. I am only wondering if, to “outsiders” (who may already a negative impression of statistics and statisticians), these methods may appear like a Rube Goldberg device. You have parameters, likelihoods, hierarchies of fixed effects, random effects, hyper-parameters, then Markov chain Monte Carlo with tuning and burn followed by long “chains” of random variables, with possible thinning for lag-correlations, concerns about convergence to an ergodic state. And after all that, newly “armed” now with a “sample” of 100,000 (or more) numbers from a mysterious posterior probability distribution you proceed to analyze these new “data” (where did the real data go? – now you have more numbers than you started with for actual data) by more methods, simple (a mean) or complex (smoothing using kernel density methods, and then pull off the mode). All OK to a suitably trained statistician, but might we be in for ridicule and misunderstanding from the public? If such a charge were leveled at us (“you guys are doing Rube Goldberg statistics”) how would we respond, given the “complaint” comes from people with little or no statistics training? Of course, such folks may not be capable of generating such a critique, but could still realize they have no idea what the statistician is doing to the data to get answers. It does us no good if the public thinks our methods are Rube Goldberg in nature.
Interesting question. I’ll respond in a few days. But in the meantime, would any of you like to give your thoughts?
My customers (business) neither know nor care about what methods I use; they come to me because, in general, they are smart enough to know they are out of their depth with their data and problem, so don't expect to understand what I do with it (otherwise they could do it themselves.)
We try to avoid the MEGO (mine eyes glazeth over) effect when dealing with customers, having seen it too often in our early years. "MCMC" etc. don't appear in our presentations or documentation outside of the technical appendix, if there is one.
But they sure do like it when I say "Based upon your data and our model, the probability that doing X made a difference is 92%" rather than this "confidence interval" stuff.
I think the public thinks statisticians compile baseball and football statistics, and some of them also think we do mathy stuff too, and that's about as much thinking about our profession as 99% of them engage in. That leaves ~3 million people who do think about our profession; hmmm, maybe I'll revise that 99% upwards.
Is the problem of Rube Goldberg techniques that they seem ridiculous to the untrained?
Surely that a layman finds a technique ridiculous is not a serious criticism. There are many things that the layman doesn't understand and won't without serious study that they nonetheless make use of. Evolution, the particle wave duality of matter, and uncertainty principle all seem ridiculous to many of the untrained. As long as they can accept the finding then statisticians shouldn't care much if folks would look down on techniques that that same public rarely sees anyway. Perhaps it is naive, but I see success in prediction and analytic understanding as the ultimate test. Most techniques that can do so get gradually wider acceptance
After dispensing with the cost benifit analysis of the strategy of just explaining "a computer is just a machine that adds, subtract and divides really fast – and that all you need to know"
I have tried out explaining getting the posterior distribution (with informative priors) as doing "nearest neighbors" in a simulated data set (posted here a couple years back)
Starting with really simple example such as a binary outcome in one or two groups – draw P (Pc and Pt) from a prior, then draw n binary outcomes given that P drawn – this gives you a simulation from the joint distribution.
Thinking of that simulation as a data set, from the nearest neighbors to your actual sample outcome you get a batch of P's drawn that led to your sample outcome (i.e. joint -> conditonal).
You can try try relate the prediction of an individual's unknown y from their x's using a data set of "similar" individuals where both the y and x are known (i.e. unknown p and known x/n in your study, predict your unknown p from the assumed joint distribution of p and x). Judgement of similiraty is crucial in both and why I mentioned it was for informative priors.
Then I "explain" MCMC as a tricky sampling method that assures the samples you draw from the joint distribution always have drawn x equal to your observed x.
Far from convinced this explanation does more good than harm – it draws appropriate smirks and giggles from those who know Bayesian methods, but is not easy for most to believe (at least initially) – and you can only do the direct sampling from the joint distribution for really simple problems.
But hopefully, it would allow someone without a lot of technical background (just a sense of prediction and similation) to grasp the logic of getting a posterior sample from an informative (or at least realistic) prior and an observed study result that could be "thought of" as being sampled from the joint.
On the other hand, there are all those other issues raised in the previous post to this one that perhaps (more importantly) need to be explained as well…
Perhaps the real question – is what do the non-statistical members of a research community need to grasp about Bayesian analysis to enable better research in that community. My guess would be the credibilty of the joint model and the various roles of various priors and the resulting salience of various interpretations of the posterior …. MCMC is just the computer adding and subtrating really fast to get the posterior.
Keith
Looking at wikipedia the definition fo a Rube Goldberg machine is one that works to solve a simple task using an overly complex mechanism. I think there are several other similarly unproductive mis-perceptions of statistics. From my experience in an applied setting, people tend to be split between those who think statistics involves plugging numbers into and pressing a button (or maybe two), and those who think statistics is a terribly boring combination of high level mathematics and magic (shockingly the fact that magic is involved doesn't excite them).
In either case, communication between the statistician and the consumers of the statistics can be a tough point that certainly isn't going to get any easier as the sophisitication of the applied methods increases.
The social model here — in terms of "statisticians" and "public" — is oversimplified.
There is a spectrum from researching statisticians through to the completely illiterate. Some way stations on that spectrum:
(a) people who apply statistics routinely as a large part of their profession but in essence wouldn't attempt to follow or wouldn't understand e.g. ongoing modern Bayesian literature (e.g. many people who work in biostatistics or medical statistics, have a Master's background, and use a more-or-less prescribed suite of methods standard in their field)
(b) people who use statistics some of the time as part of their research (e.g. many scientists): naturally many of these people may be way out-of-date in terms of such topics
(c) people who once did some introductory statistics in U.S. college or equivalent
(d) NY Times readers or equivalent with good high school or college-level education
etc. etc.
In short: what public are you talking about?
But the question is a good one.
I think there is a real problem. What is in basic introductory statistics courses is in a strong sense way too complicated already for much of the student body, despite several waves of re-orientation aimed at making the material more exciting, palatable, relevant, etc. How much of what is mentioned here could possibly filter down below moderately advanced graduate courses?
I agree with TheOneEyedMan.
Current models for ranking search results or doing spam filtering are a good example. They're quite Rube-Goldberg-esque. Given that they work fairly well, no one's mocking their creators for using methods that are too complex.
Isn't there also a chance of being mocked for a model that's overly simplistic?
Considering the ignorance that some people possess and the way they seem to wave it around like a badge of honor*, I think their opinion is important when such people possess a certain level of authority or sway on public opinion. In general I think it is important that people be able to understand the things we do.
My impression is that most laymen look at statistics with a certain sense of awe, but it's also mixed with Twain's quip about damn lies and statistics. I agree with One-eye that prediction and ability to explain a phenomenon are important and should be the benchmark of success. This is part of the power and beauty of statistics. But it's also important to explain the device that helps us to understand the phenomenon. There's a point at which non-technical people also need an explanation, and the simpler and clearer the better. It doesn't need to be a deeply theoretical picture.
If statisticians decide not to present a clear picture, then we deserve to be called gnostics, and we'd also deserve whatever mistrust might develop between ourselves and laymen.
*As an example, I think of the 2000 Census adjustments that caused controversy.
I dont think the public has even the slightest awareness of the differences among statistical approaches; even highly educated folk. To them, it seems like numerical voodoo, and their lot is cast either with the group that is suspicious of any statistical estimate or model, or with the group that mostly trusts that trained statisticians know what they are doing. Very few people that I know who arent academics (lawyers, business owners, and the like) have any opinion on statistical methods whatsoever.
But, for those who do choose to scratch the surface, they should have no greater cause for confusion than with frequentist techniques: "tell me once more what a 95% confidence interval means?" or "how come alpha=0.05 is significant, but 0.06 is not?" Try explaining those to Jane Q. Public!
A Rube Goldberg device isn't a black box. There are normally steps that people can follow, if only to conclude that it is a ludicrous way to do something simple. But it would be a rare person indeed that could follow the complex path taken by an advanced statistical procedure only to conclude that it was popping out an answer that could have been found with a simpler statistical technique.
Just the same, someone trained in advanced data analysis could challenge the complexity of a statistical technique, claiming a simpler approach were possible. And they should. In an academic environment you can defend yourself by saying that it's a new technique, that could lead to other developments. The simpler, possibly more well established, method can be used to validate the more complex one. Presumably the complex technique would be appropriate in other circumstances. In a more applied setting defending one self may prove more difficult, unless there is good reason to use the more complex method. In which case the statistician should be prepared to answer such questions.
Ultimately it comes down to results. There must be compelling evidence, appropriate figures and explanations, and theory to back up statistical results. We should be questioned on the methods we employ, but I believe it has to be taken on a case by case basis. I can't speak in defence of MCMC-based Bayesian methods, but I wonder how we would defend statistics in general from such criticism.
Let's be Bayesean about this.
Simply put, in most of our lives the following rule works: if I don't understand it, it's more likely to be wrong.
This rule works well when investing your money, buying a fancy piece of consumer electronics, evaluating your teen's rationale for a late night out, and a host of other circumstances.
Should it be any different for statistical analysis? Is it really a good idea for a policymaker to act on a finding if they don't really understand the analysis — not to the level of being able to do it, but to the level of being able to understand it?
In addition, often the more complicated the technique, the easier it is to use it without fully understanding it.
If bad statistical models were a capital offense, not even Texas could execute people fast enough.
Tip: When talking about sampling, never use the word "random". Most people associate that word with "meaningless" – the most unscientific thing one could think of.
LemmusLemmus humorously (?) suggests not calling it "random" sampling.
Systematic sampling works well — both as a technique and as a term.
Re: Rube Goldberg statistics. It's amazing how far you can get by just taking the sample mean!