Since we’re on the topic of nonreplicable research . . . see here (link from here) for a story of a survey that’s so bad that the people who did it won’t say how they did it.
I know too many cases where people screwed up in a survey when they were actually trying to get the right answer, for me to trust any report of a survey that doesn’t say what they did.
I’m reminded of this survey which may well have been based on a sample of size 6 (again, the people who did it refused to release any description of methodology).
Of course, even if they tell you about the sample, we all know that the choice of questions can make huge differences. Jon Krosnick at Stanford has done some interesting surveys, but also:
"His questionnaire design work has illuminated the cognitive and social processes that unfold between researcher and respondent when the latter are asked to answer questions, and his on-going review of 100 years worth of scholarly research on the topic has yielded a set of guidelines for the optimal design of questionnaires to maximize reliability and validity. His recent work in survey methodology has explored the impact of mode of data collection (e.g., face-to-face, telephone, Internet) on response accuracy and the impact of survey response rates on substantive results. "
He gave a lively talk on this about a year ago, with absolutely cringeworthy examples of bad questions.
John:
Yes, "how they did it" includes question wording, mode of interview, etc., not merely the sampling design.
On this topic, readers of this blog may be amused by the catfight in the comments section of this posting over how and whether to survey diners for restaurant reviews. I particularly enjoyed the efforts of the author in question to namecheck the Freakonomics folks and Nate Silver in support of his position.
I had some great tacos in Minneapolis.
The Burnham et al. (2006) survey on deaths in the Iraq war (that has been discussed on this blog) is another example of a survey this is not really a survey because they don't tell you how they do it. The Standards Committee of the American Association for Public Opinion Research (AAPOR) issued a rare censure making this point official. See:
http://www.aapor.org/Content/NavigationMenu/Press…
and
http://www.aapor.org/uploads/AAPOR_Press_Releases…
The AAPOR Standards Committee also issued a complaint about the polling firm Strategic Vision:
http://www.aapor.org/Content/NavigationMenu/Press…
leading to a court settlement with its client, the Daily Kos. Unfortunately there seems to be a gag rule surrounding this settlement:
http://www.huffingtonpost.com/2011/05/27/daily-ko…
Mike Spagat