Chris Masse pointed me to this blog by Panos Ipeirotis, who argues that some online prediction markets give probabilities that are too good to be true:
If your markets do not fail, they are a failure! Let’s see the set of claims from HubDub, who believe that they nailed the results, in contrast to InTrade that missed the “best actor” award. So here are the results for the 6 major categories from the two exchanges, together with the probabilities for the frontrunners:
Category Hubdub InTrade Best Picture 98% 90% Best Director 76% 90% Best Actor 63% 70.0% (wrong) Best Actress 87% 85% Best Sup Actor 100% 95% Best Sup Actress 64% 58.8% So the question is: How many contracts should they get correctly to claim good accuracy?
The knee-jerk answer is “all of them”. Unfortunately, it is incorrect as well. According to the reported numbers, for HubDub the probability of getting all the answers correctly is 0.98*0.76*0.63*0.87*1.0*0.64=0.26. For InTrade, the corresponding probability is 0.287.
Is there a more likely outcome? Yes, for HubDub, according to their own numbers, they should have picked correctly 5 out of 6 frontrunners, with probability 0.42. For InTrade, they should have picked correctly 5 out of 6 frontrunners with probability 0.43. . . . Guessing all the frontrunners correctly is something to brag about ONLY if the reported confidences are high enough. If they are not and you get them all correctly, then the markets have biases and are NOT accurate.
I assume that Ipeirotis realizes that events with probability .29 can occur by chance–so this can’t really be taken as a criticism of Hubbub or Intrade in this instance.
It has been discussed, however, that Intrade can be underconfident with probabilities near 0 or 1–for example, it was giving John McCain a 4% chance of winning D.C. Presumably this has something to do with transaction costs and the impossibility of actually making money off these “Obama will win D.C.” sorts of bets.
More generally, I have occasionally noticed validations of Bayesian analyses in which way more than half of the 50% posterior intervals include the true value. Invariably this turns out to be a problem in the model or the computation.
Andrew,
Of course I realize that it is possible to get 6/6 with some probability.
What I want to point out is that always celebrating the win of the most likely event (MLE) is a self-contradicting: Having all the markets pick correctly the winner is *not* the most likely event.
I am trying to get people to understand that the goal of prediction markets is NOT to "nail" every single market. Apparently, a very hard task.
So, I will take the extreme opposite position just to get people to read and discuss :-)
For longer-term bets (like the 2008 election) liquidity preferences are a factor too. Someone who has bet on an event at 2/1 is often indifferent to losing a few pence on the pound by laying off his bet, particularly when he has other high-return bets he wants to make with that money.