Nate’s got the goods. Some questions from a recent Fox News poll:
Do you think Barack Obama’s travel and speaking schedule makes him look more like he is a candidate on the campaign trail or more like he is the president of the United States?
Do you think President Obama apologizes too much to the rest of the world for past U.S. policies?
Do you think the Obama administration is proposing more government spending than American taxpayers can afford, or not?
Do you think the size of the national debt is so large it is hurting the future of the country?
Would you rather: [ROTATE OPTIONS 1 and 2]
Cut spending now so future generations don’t have to pay
Keep spending at current levels and let future generations pay
These would make a great example for your statistics classes, I think.
Here’s another one that amused me (it came after the health-care questions so Nate didn’t include it in his list):
Who do you think should have more power under the law — environmental activist groups or individual property owners? (ROTATE)
In the comments, feel free to link to any survey questions you find with a left-wing bias. (For example, a left-wing equivalent of the last question above might be, “Who do you think should have more power under the law — individual citizens or multinational corporations that pollute?”)
Polling on immigration is frequently biased by how the questions are phrased.
I don't follow polls so I have nothing to offer from a left-biased poll (no doubt one exists though), but some of those questions don't even make sense or don't even accurately reflect the situation.
Like the question #3 in your post. The answer should read 100% yes, because the answer is without question yes. The United States can easily afford to pay for just about any government expansion we desire. We can pay for it the same way every government does: taxes; or if you are a crappy government: seigniorage. What the question should be asking is if you believe that the level of taxation to fund such programs is reasonable, since it's obviously possible that we can. Wouldn't a superior question be something to the effect "Do you think the Obama administration is proposing more government spending than can be reasonably paid for in taxes?" The spending question is also deeply misleading, I think, since if you phrase the question to reflect the real-world situation you'll get a different answer. A lot of people think we can trim "fraud, abuse, and earmarks" and suddenly our fiscal situation would be perfect perfect, which is just incredibly dumb. The only problems we have our medicare / social security / defense (but much less defense), at least in the long-term. If you were to ask should we purge Medicare spending or purge defense spending or purge social security spending, I think you'd find most would be for keeping spending at current levels. I feel like polls should accurately reflect the real options we have on the table, not vague options like "cut spending."
Also, that final question is just ridiculous.
Some of the Fox News poll questions are indeed strange and argumentative. Nate Silver (who has his own bias) claims these questions taint the whole poll, and thus we can't trust the responses to the neutrally worded questions. That's certainly possible, but where is the evidence? Surely this question has been investigated. Silver just seems to assert his hypothesis as fact.
Why not randomize the question order over the course of the poll so each person polled gets a different order? I would think this might compensate for a
"fatigue effect," and any tainting.
Again surely there must be research on question order, it would be easy to do and fairly important.
Immigration poll questions are often phrased in such a way that if you really thought about it logically, there would be only one reasonable response.
Fortunately, nobody thinks logically, so the poll results are semi-useful.
I find it amusing that they're bothering to rotate response options. I mean, once you've written those questions, is the potential bias of order effects really a big concern?
Ted: Exactly.
Zarkov: Yes, such research has been done, I believe. Also, Nate does not merely "assert his hypothesis as fact"; he leads off by discussing some systematic differences between the results of Fox compared to other polls.
Steve: Often the claim is made, perhaps reasonably, that even if biased poll questions corrupt the average survey response, they can still be used to learn about time trends and difference between groups.
Aaron: According to Nate, this is a professional survey organization. So maybe the politicos give them the question ordering and then the pros take it from there and do their best. I wonder what sorts of internal discussions they've had about these sorts of wacky questions.
Andrew:
Nate goes beyond simply asserting Fox News polls on the healthcare question show a systematic difference with other polls. He advances a specific cause: question order. He's telling us that the questions that appear before the healthcare question somehow induce some respondents to change their answers. While I think this a plausible hypothesis, he offers no evidence for his "contamination effect."
I worked on errors in survey sampling in a physical science context. Let's suppose every member of a target population is a "0" or a "1," and we want to estimate the finite population total from a sample. But let's say that sometimes a "0" gets recorded as a "1" and vice versa. The former error will falsely increase the population total estimate while the latter will reduce it. If you know what I call the "conversion probabilities," P[0->1] and P[1->0], you can calculate how the estimate gets distorted. In one case the results are completely unintuitive. There are critical values for the conversion probabilities that will cause the survey to produce complete nonsense. I wouldn't want my survey to be anywhere near the critical values. The answers are a little complicated involving confluent hypergeometric functions, but they simplify with appropriate approximations.
Now Nate is telling us that question order, and the type of question can affect the conversion probabilities. This is why I'm interested. I never published my stuff because I thought surely someone must have done it before.
Within extremely broad limits, a professional survey organization [i.e. they get paid] will ask what the paying client wants asked in the way they want it asked.
One good example of this is the Zogby internet polling. The sampling scheme is curious (you suggest other people) and the questions look like each 6th grader had to submit a page.
Zarkov: There's been a lot of work on survey biases such as question wording effects, and I think it's generally believed (with good reason) that this sort of contamination can indeed occur.
This dicussion reminds me of a scene from the British political comedy Yes Prime Minister:
http://www.youtube.com/watch?v=2yhN1IDLQjo&NR=1
@Zarkov: Nate has gone over the subject numerous times. It would be somewhat repetitive for his audience were he to go into it again.
Zarkov,
I'd be interested to see those results if you ever write them up, even as an unpublished e-print.
Isher:
Nate should then provide a link to a prior thread on this topic. I'm not a regular reader of 538. The very few times I've visited the site, I found the discussion to be low quality.
NU:
How do I use e-print? If I find that no one has done this before, I'll write it up for a journal with some extensions I have in mind.