AT writes:

I’ve got a count-based data set with a lot of zeroes present. I’m using zero-inflated modeling to capture the shape, and I want to test goodness-of-fit from both ends — under- and overfitting. I’ve read your 1996 paper with XL and Hal Stern which recommends a “discrepancy measure” as being a good quantity to calculate with posterior predictive data. The main suggestion there was to use a chi-square statistic, but I’m sure this would be inappropriate in this case given that the zero cases would drive the entire statistic (and breaking the minimum-cell-size rule for the chi-square about 500 times in the process.) I suppose we could correct for this by doing the square-root trick to stabilize variance, but that still doesn’t seem like it would resolve the problem with the zeroes. Any thoughts as to how to find a good discrepancy measure to check?

My generic response is that we always want the test summaries to relate to the substantive questions of interest. In this case, I don’t have the context but I can make some quick suggestions, such as to create two test summaries: (a) the percentage of zeroes, and (b) some summary of the fit ot the counts when they are not zero.

The so-called minimum cell size rule is irrelevant, since you can compute the reference distribution directly using simulation. And issues such as stabilizing variance are not particularly relevant either, except inasmuch as they allow your test to more accurately capture the aspects of the data that are important for you to fit with your model.