Kent Holsinger writes:
I’m fitting a moderately complicated hierarchical model including a structural equation model with latent variables to plant traits measured in a greenhouse.
My interest focuses both on the regression coefficients within the SEM and on additional regression coefficients relating relating to covariates that influence the manifest variables outside the SEM. In addition, I’ve also included some random effects — bins — on the manifest variable to account for the fact that some of the plants had to be grouped together for experimental purposes. I’m not interested in the “bin” effect, per se. It’s just there as an experimental control. But the magnitude of some of the regression coefficients differs substantially between the models with and without bins, which is a little troubling.
If we use DIC to compare the “with bins” and “without bins” models, this is what we get:
From the “with bins” run:
Dbar: 9752.031
Dhat: 4630.831
pD: 5121.2
DIC: 14873.231
From the “without bins” run:
Dbar: 9098.022
Dhat: 5348.022
pD: 3750.0
DIC: 12848.022In other words, DIC would suggest that the “without bins” model is “better”. Nonetheless, I am inclined to report the regression coefficients from the “with bins” model, (a) because it’s the one where any bin effect is properly controlled for and (b) because it does fit the data substantially better (in the sense that the deviance at the posterior mean — Dhat — is much smaller for the “with bins” model than for the “without bins” model). Does that make sense to you? If so, can you suggest a paper we could refer to to justify that choice? Biological reviewers may not accept my reasoning unless we can refer to some statistical “authority figure”. Would it be reasonable simply to cite pp. 244-246 of Gelman & Hill?
My reply: Yes, of course you can cite Gelman and Hill. You can’t get much more authoritative than that! Seriously, though, I have no experience with structural equation models and so I can only speak in generalities. So, to start with, please round all DIC’s to the nearest integers. Anything beyond that is basically meaningless. Beyond this, I’m surprised that the “with bins” model is much worse in DIC. It’s a hierarchical model, so if the data really want it to be “without bins,” I’d think the bin-level variance would be estimated at close to zero, making the “with” and “without bins” models essentially identical. I don’t see how you get the 2000 difference in DIC. So, there’s something going on here that I don’t understand.
P.S. I’m posting this today because somebody just sent me a mean message saying that my blog wasn’t very technical. He wasn’t trying to be mean, but he hurt my feelings anyway. My blog can too be technical! And what about this, from a few days ago? Huh? Huh?? So there!
The values of pD are very large in both models. This suggests to me that there are some highly influential observations, although it is impossible to tell without knowing the sample size.
In this paper I showed that DIC is an approximation to a penalized loss function, which holds when you can drop individual observations without perturbing the posterior distribution. A key statistic is pD/N where N is the sample size. This must be very much smaller than 1, otherwise the DIC is not reliable.
Was the title of this post an intentional double entendre?
Martyn: Thanks for providing some backing to my intuition.
William: I don't see the double entendre here. Unless you're saying that any use of the word DIC is a double entendre.
Where I live, calling something a "dick question" is a rude way of saying that the question is insincere and self-serving. This question doesn't seem to be that, but I honestly don't know enough statistics to know that. If somebody had asked you a question about DIC in way that was insincere and self-serving, titling the post "DIC question" would have been brilliantly funny (in the same vein that the Governor of California used recently with plausible deniability…)
You, unlike Mr. Schwarzenegger, don't seem to be the type to actually respond that way, so I was momentarily confused.
I'd never heard that expression.
The variance component may have a wide credible interval, though. I'm sure you know the shape – very right skewed. I don't know Kent's data, but this is typical with biological data sets (well, outside of humans, of course). This will obviously pull up both the mean deviance, and pD (if it's being estimated from the posterior means).