James O’Brien writes,

Just wondering if you had any thoughts on testing a random intercepts model in multilevel logistic regression. In my present study, the results of a Wald test (which Twisk, for example, suggests as a kind of approximation for testing whether parameter variance is effectively zero), produce a p-observed of .06.

I’ve noticed in the MLWin manual as well, they argue that p<.1 for the Wald is evidence that slopes and intercepts (in that particular case) indeed vary. Are they perhaps willing to relax the cut-off because of the approximate nature of the test? Am I perhaps missing something? Perhaps I need to go back to MLWin and do some bootstrapping.

My reply: I’ve been known to bootstrap (see my 1992 paper in the Journal of Cerebral Blood Flow and Metabolism, and my recent paper with Alessandra analyzing the storable votes data), but in this case I don’t think you have to go to that level of effort.

The short answer is that these true variances are never zero (at least, not in social science applications); a non-significant test doesn’t mean the variance is zero, it just means that the data are consistent with the zero variance model. So, if you want, you can keep the variance component in the model and ignore the test. Conversely, if the variance is low, it might not hurt you much to just exclude it from the model (i.e., to set it to zero). It depends what you’re doing. If you’re just including this variance component to help estimate some other parameter in the model, you can probably just get rid of it, but if you’re specifically trying to compare particular coefficients here, or to make predictions for new groups, you better keep it in. In that case, you might want to do fully Bayes (something like the 8-schools model) so you’re not conditioning on a particular imperfect estimate of that variance parameter.

This issue actually comes up a lot. Fortunately, it’s when the variance parameter is least important (when it could in practice be replaced by zero) that it’s trickiest to estimate. A rare bit of good news in statistical inference.