Regular readers will know the importance I attach to model checking: to the statistical paradigm in which we take a model seriously, follow its implications, and then look carefully for places where these implications don’t make sense, thus revealing problems with the model, which can then trace backwards to understand where your assumptions went wrong.
This sort of reasoning can be done qualitatively also. From Daniel Drezner, here’s a fun example, an analysis of a recent political bestseller:
I [Drezner] hereby retract any and all enthusiasm for Game Change– because I don’t know which parts of it are true and which parts are not. . . . It was on page 89 that I began to wonder just how much Game Change’s authors double-checked their sources. This section of the book recounts entertainment mogul David Geffen’s “break” with Hillary Clinton’s presidential campaign:
The reaction to the column stunned Geffen. Beseiged by interview requests, he put out a statement saying Dowd had quoted him accurately. Some of Geffen’s friends in Hollywood expressed disbelief. Warren Beatty told him, She’s going to be president of the United States–you must be nuts to have done this. But many more congratulated Geffen for having the courage to say what everyone else was thinking but was too afraid to put on the record. They said he’d made them feel safer openly supporting or donating to Obama. Soon after, when Geffen visited New York, people in cars on Madison Avenue beeped their horns and gave him the thumbs-up as he walked down the street (emphasis added [by Drezner]).
A self-refuting sentence indeed. Don’t these guys have an editor? This reminds me of our recent discussion of the economics of fact checking.
Another hypothesis is that John Heilemann and Mark Halperin–the authors of Game Change–realized all along that the thumbs-up-on-Madison-Avenue story was implausible, but they felt that it was a good quote to include in order to give a sense of where Geffen was coming from. From this perspective, it should be obvious to the reader that the sentence beginning “Soon after, when Geffen visited New York” was a Geffen quote, nothing more and nothing less. In a book based on interviews, it would just be too awkward to explicitly identify each quotation as, for example, writing, “Geffen told us that soon after he visited New York, people in cars . . .” Sure, that latter version would be more accurate but would disrupt the flow.
Similar reasoning might explain or excuse David Halberstam’s notorious errors in his baseball book that were noted by Bill James: Halberstam’s goal was not to convey what happened but rather to convey the memories of key participants. Similarly, maybe the point of Game Change is to tell us what people recall, not what was actually happening. An oral history presented in narrative form.
P.S. For more on model checking from a Bayesian statistical perspective, see chapter 6 of Bayesian Data Analysis or this article. Or, if you prefer it in French, this.
Beeping horns in Manhattan? Believable.
Thumbs-up? Those weren't thumbs.
Perhaps this is just the Hollywood everything-must-relate-to-the-story version of a pedestrian walk through a crowded urban area.
Always – where your assumptions went _most_ wrong ;-)
This (assumptions) always being wrong is perhaps very awkward to get into our writing – as it is to get into our heads.
Of course, that is what Andrew meant.
K
I was just reading the terrorist detection section of Freakonomics 2 and a similar sentence jumped at me. They were describing the British guy who claimed to have this super duper algorithm. He apparently went to bed dreaming of (and I think they even said sifting through) billions and billions of data points. Reminds me of the scene in the John Nash movie but hardly believable. Can writers of non-fiction use "dramatic license"?
"see chapter 6 of Bayesian Data Analysis"
Please try to produce your book for Kindle.
I believe that Mark Halperin avoided quotations in the book because all conversations were officially "on deep background". The whole thing is ridiculous. We're not supposed to know that he interviewed Geffen at all.
One really obnoxious effect is when they state that XXX as a fact, rather than "we were told that XXX". Another issue is that some of the sources consider themselves to have been burned, because their "deep background" conversations were essentially quoted. See Harry Reid on negro dialects, who did .
Kaiser:
My theory (elaborated upon here) on the lapses in the Freaknomics blog and in Freakonomics 2 is that it was a division of labor that went awry. Dubner's job was to tell stories, and he figured that Levitt would check that the statistics all made sense. But the statistical work was all done by Levitt's friends, and so Levitt relied on his friends to have checked their own claims. The result is: lots of claims that didn't get checked. Ironically, I suspect that if Dubner had been the sole author of the book, he might've been more careful. Having Levitt as a coauthor perhaps gave him a false sense of security, and perhaps Levitt is just too much of a nice guy and trusts his friends too much.
Alex:
I know what you're saying, but I still wonder whether Heilemann and Halperin were working under some journalistic principle that their highest duty was not to report objective truth about what happened, but to report truthfully what people said to them.
From this perspective, the reporters would view themselves not as investigative journalists who, Sherlock Holmes-like, put together the pieces into a coherent narrative, but rather as interviewers who can collect the statements of the key players.
The truth of what is said, or the truth of what happened. Didn't Wittgenstein or Tarski or Quine or somebody write about this stuff?
Wittgenstein or others certainly did – recalling from my undergraduate course, Wittgenstein's "picture theory of language" was found, even by him, to be deficient for many reasons and perhaps importantly (to him) for not considering intentionality.
But it does bother me we people quote things they know are false because it supports their case — but does not put them at risk for making the error – but they are propagating it.
For instance in Richard Doll's Fisher Lecture (2001) he mentioned the crticism that Fisher was best at dealing with small single data sets and not good with multiple studies.
When I pointed out that many of Fisher's publication explictly deal with multiple studies and multiple data sets – he simply said that he was somewhat aware of that – but he was just quoting what others said of Fisher.
K
p.s. Fisher and Doll had a nasty argument about smoking and lung cancer – and there is a post or two about that on this blog a few years ago.
As zbicyclist says, "those weren't thumbs".
This is an old joke, I think you (and Drezner) missed it. It's similar to Yakov Smirnoff going to New York and thinking everyone is suddenly his buddy.