Mike Cohen writes:

The recent events in Egypt raise an interesting statistical question. It is of course common for news stations like CNN to interview various officials and policy experts to find out what is likely to happen next. The obvious response of people like us is why ask such people when they didn’t foresee a month ago that these dynamic events were about to happen. One would instead like to hear from those experts that did predict that something was about to happen in Tunisia, and Egypt, and Jordan, and maybe Yemen, etc. Well, are there such people? My friend Bob Burton says that of course one can find such people in the sense that they made such predictions, but that is like finding counties that have voted for the President in the last five elections, big deal, or psychics that predicted the last assassination, again big deal. There is a good deal of truth in that. However, it seems like we do a little better. There are two points to make. First, there is an information theory aspect to how impressed we should be by predictions that are made. In the following I am glossing over a very important point, which is how specific the predictions are, and that is worth thinking a lot about. But speaking loosely, for a dichotomous question with both sides close to equal a priori, making a prediction that turns out to be correct is no big deal. But predicting the date and manner in which some political event would occur, such as a change in government well in advance, when the general a priori view was that nothing was going on, would be extremely impressive. So question one is can one measure how impressive a prediction is, to help us understand who our experts really are, and what experts are just guessing well? Second, does Efron’s work 40 years ago on assessing whether Bode’s Law has a physical basis relate to this question? Maybe not, but Efron was looking to see whether the closeness of a hypothesis and data was such that one could say that it was not likely to have occurred by chance, and there seems to be something similar going on when comparing a number of such predictions by someone and their closeness to later events. Any thoughts?

I replied as follows:

My quick thought is that any expert forecasts should be evaluated not in a vacuum but in the context of their qualitative models. Regarding Egypt, you might be interested in Daniel Drezner’s remarks about political scientists’ work in that area.

To which Cohen pointed out that an expert could predict that something was going to happen without having an explicit theory that they were operating from and that would still be of interest, though less so, than if they had put forward how they arrived at such predictions.