Don’t look at just one poll number–unless you really know what you’re doing!

Here’s a good one if you want to tell your students about question wording bias. It’s fun because the data are all on the web–the research is something that students could do on their own–if they know what to look for. Another win for Google.

Here’s the story. I found the following graph on the front page of the American Enterprise Institute, a well-known D.C. think tank:

aei.png

My first thought was that they should replace this graph by a time series, which would show so much more information. I did a web search and, indeed, looking at a broad range of poll questions over time gives us a much richer perspective on public opinion about Afghanistan than is revealed in the above graph.

I did a quick google search (“polling report afghanistan”) and found this. The quick summary is that roughly 40% of Americans favor the Afghan war (down from about 50% from 2006 through early 2009).

The Polling Report page also features the Quninipiac poll featured in the above graph; here it reports that, as of July 2010, 48% think the U.S. is “doing the right thing” by fighting the war in Afghanistan and 43% think the U.S. should “not be involved.” This phrasing seems to elicit more support–I guess people don’t want to think that the U.S. is not doing the right thing.

OK, so we have 40% support, or maybe 48% support . . . how did the AEI get the 58% support highlighted on its graph?

Searching the Polling Report page for the word “worthwhile,” I find this from the Quinnipiac poll:

“Do you think eliminating the threat from terrorists operating from Afghanistan is a worthwhile goal for American troops to fight and possibly die for or not?”

“Is worthwhile”: 59%
“Is not worthwhile”: 34%

But is the AEI correct to identify “a worthwhile goal” with the statement that “fighting in Afghanistan is worthwhile”? Maybe not. There’s a difference between a goal and an action. Consider the next question from that Q poll:

“Do you think the United States will be successful in eliminating the threat from terrorists operating from Afghanistan or not?”

“Will be successful”: 35%
“Will not be successful”: 55%

It looks to me that, out of the 60% of the people who think that eliminating the terrorist threat in Afghanistan is a worthy goal, about 20% (that is, one-third of the 60%) don’t see the war as a good way to do that.

I think the AEI graph is misleading–although maybe not intentionally so. This could be a good example in your political science or statistics classes of the peril of pulling out just one number without looking at the broader range of survey questions. And it also suggests some aspects of public opinion research that might be useful, even to political practitioners such as Steven Hayward, who maybe should spent less time in his classes making fun of the American Political Science Review and more time teaching his students about question-wording effects.

P.S. I am not commenting in any way on the merits of the Afghan war. I’m just trying to report public opinion accurately. Which, as can be seen here, is much easier to do if we take advantage of the wealth of poll information directly available on the web.

P.P.S. I re-checked at the AEI site and the graph shown above is no longer there. So maybe someone noticed the problem and took it down.