Bernard Guerrero asks what I think of this. My response is that people can be irrational all the time–let’s face it, we’re a bunch of animals. Voters can have incoherent preferences (e.g., more services but less taxes), consumers can make mistakes (buying that brand-name $40,000 car and then being upset that they have no money left), forecasters can make mistakes (even setting aside “moral hazard” settings, there are lots of notorious problems such as people attaching insufficient probability to the “all else” category).
Arima models etc. can be overrated–lots of people seem to think these are the only models out there. Cavan Reilly has a fun example–chapter 27 in my book with Meng–of a 6-parameter predator-prey model that way outperforms standard time series models (with 11 or more parameters) in forecasting the famous Canadian lynx series. So I’m not surprised that Arimas can be beaten.
I agree with Bernard that you’d want to know where the survey forecasts come from. The surveys themselves are of forecasts. (This is different than the familiar use of surveys of forthcoming elections, where people are asked whom they would vote for if the election were held today. The Ang et al. paper is using surveys where people are explicitly asked to forecast.) It does sound like a classic “wisdom of crowds” averaging.
P.S. Two of the authors are at Columbia. I haven’t met them. Perhaps they can speak in our quantitative social science seminar in the fall.
The wisdom of crowds is always very appealing. On the downside there exist social phenomenons as Groupthink. These may lead experts to restrain their true opinion, because they think the majority favors something else. Yes, we are animals.
Fortunately, computers are immune to this, so there is hope for the wisdom of crowds of algorithms, such as in multi-view learning.
"The Wisdom of Crowds" surveys a few methods of aggregating information, and the necessary conditions for success rules out modern political processes. Independence, for example, is essentially absent, and the voting methods of public elections are the ones most likely to produce aggregation pathologies.
Of course, what is a non-rational voter anyway? Have you ever seen a scientific professional give his religion the imprimatur of thermodynamics when he says that evolution is impossible because it violates "entropy," apparently hoping his listeners will confuse "entropy" with entropy?
If an economist talks about rationality, then she is talking about a logical binary relation of preferences that is defined by completeness and transitivity, or she is talking about the necessary and sufficient pattern of choices that go with said preferences. If she is talking about anything else, then she is talking about "rationality," not rationality, and has the same expert authority as the creationist-scientist talking about "entropy" viz. evolution.
(And maintain the sort of skepticism for any economist who talks a lot about free markets as you would for a geologist who talks about diluvial geology.)
What the writer fails to understand is that the wisdom of crowds comes from the properties of the average of the judges, not from the properties of average judges. The body of literature, much of which was produced in the early 1900s, points to a diverse crowd of people who's opinions are part accurate information and part error. If the opinions of the individuals bracket the truth — some err on one side, some on the other — then those error components cancel out, making the average of judgments more accurate. But you've indicated that you know all that already, so suffice it to say that a mob of Americans may more accurately forecast next year's GDP, but an army of Gollums would be no closer to guessing what's in a hobbit's pocket.
It's interesting that because of the nature of the problem, crowds are better at guessing regardless of the randomness of the sampling technique. The survey to forecast next year's GDP is not an attempt to accurately describe the public's opinion; instead, the object is to get guesses sufficiently diverse and independent to capture the bracketing phenomenon. If planners would survey to get facts rather than survey to measure opinion, our land-use decisions would probably be much better informed.
Caveat lector, of course. Except for the bit about rationality: no physicist would talk of impetus for momentum, but for some reason economists will talk of grub-more-money for rationality. It's very strange.