A colleague heard this and commented that I considered various alternative possibilities, for example ten straight heads or ten straight tails, but not other possibilities such as alternation between heads and tails. He also wondered if I was too confident in saying I could correctly identify the true and fake sequences and suggested I could report my posterior probability of getting the correct identification.

My reply:

The interview was taped, cut, and put back together before airing. The main effect of this was to make me sound more coherent and poised than I actually am, but a couple of points did get lost.

First, I actually did mention situations when I’ve guessed wrong, either because the real coin flips happened to have a lot of alternation, or because the kids creating the fake flips actually knew the secret and created long runs.

Second, even beyond it being a 1/250 chance of 10 straight heads, 10 straight tails, 10 AFC coin flip wins, or 10 NFC coin flip wins, there have also been over 30 sequences of 10 super bowls–and 30 chances of 1/250 shot isn’t so extreme at all. I actually did think of the case of alternation of heads/tails, or wins/losses, but it seemed to me that such a pattern might not be noticed as exceptional, so I didn’t mention it.

So, yes, I agree that this sort of classical significance-testing reasoning includes a lot of uncomfortable psychologizing and speculation, and indeed it’s a motivation for Bayesian inference. But really I think of this more of a probability example than a statistics example, and typically the discrimination between real and fake sequences is clear enough.

I’d be happy to report a posterior probability–the ideal approach would be actually do the demo a few hundred times on various classrooms (you can have students work in small groups, rather than simply divide the class in half, so as to get several fake sequences per class–but, for reasons discussed in our book, I wouldn’t have the students do the sequences individually), and when I get a pair of sequences, one real and one fake, give my subjective judgment as to which is the fake one, along with a numerical measure of my strength of certainty (e.g., on a 1-10 scale). Then with a few hundred examples, I could easily fit a model to calibrate my certainties onto a probability scale.

Finally, I didn’t get a chance to credit Phil Stark, who told Deb and me about this demo. I did, however, tell them that the example is well known in statistics teaching.

P.S. The demo is described in this book and also in this article.