Jeronimo pointed out this analysis by a bunch of statisticians comparing the 2004 exit polls with election results. The report (by Josh Mitteldorf, Kathy Dopp, and several others) claim an “absence of any statistically-plausible explanation for the discrepancy between Edison/Mitofsky’s exit poll data and the official presidential vote tally” and hence suggest that the vote itself may have been rigged.
Mittledorf et al. (hereafter, “US Count”) present a number of arguments based on the results in the Edison/Mitofsky report that leave me intrigued but not convinced.
1. US Count starts with a histogram (on page 6 of their report) demonstrating that Bush outperformed the exit polls in most of the states. US Count then perform some implies some p-value calculations showing how unlikely this would be “less than 1 in 10,000,000” if the state errors were independent. But the errors are clearly a national phenomenon, so a calculation based on independent state errors misses the point. The real issue, as US Count recognizes elsewhere in its report, is: How plausible is it that Kerry voters responded to exit polls at a 6% higher rate than Bush voters, as would be needed to explain the errors?
2. US Count make various calculations of error rates in different precincts. These are interesting–at the very least, I don’t think the patterns they find should be occurring if the poll is executed according to plan–but I don’t see how they rule out an overall higher rate of response by Kerry than by Bush voters.
3. US Count notes that the exit polls predicted the Senate races better than the Presidential races in the states. Here are the errors (just for the 32 states in the data with Senate races):
(By the way, I give all error rates in terms of Democrats’ share of the vote. Edison/Mitofsky gave errors in vote margin, which I found confusing. My numbers are just theirs divided by 2.)
Anyway, there is definitely something going on, but again it appears to be a national phenomenon. I’m not quite sure what sort of hypothesis of “cheating” would explain this.
Considering the US Votes hypothesis more carefully, it makes sense to look at the Edison/Mitofsky “composite estimates,” which combine the exit poll with a “Prior Estimate, which is based upon analysis of the available pre-election surveys in each state.” Unsurprisingly, these composite estimates are better (see page 20 of the Edison/Mitofsky report). And in assessing the hypothesis that the polls are OK but the votes were rigged, it makes sense to use these better estimates as a baseline.
Here are the errors in the Presidential and Senate vote from the composite predictions (exit polls combined with pre-election polls):
Discrepancies have changed but are still there. One hypothesis for the differences between Presidential and Senate error, considered and dismissed by US Votes, is split-ticket voting. In fact, though, the states with more split-ticket voting (as crudely measured by the absolute difference between Democratic Senatorial and Presidential vote shares) do show bigger absolute differences between Senate and Presidential errors.
4. Discrepancies are lowest with paper ballots and higher with election machines. I don’t know that there are any reasonable hypotheses of fraud occurring with all machines (mechanical, touch screen, punch cards, optical scan), so I’m inclined to agree with Edison/Mitofsky that these differences can be better explained by rural/urban and other differences between precincts with different types of polling equipment.
5. The exit poll data do show some strange patterns, though. Let’s go back to the state-by-state errors in the Presidential vote for the 49 states in the data. Here’s a plot of the state error vs. Kerry’s vote share:
What gives with the negative slope (which is, just slightly, “statistically significant”)? This is not what you’d expect to see if the poll discrepancies are entirely due to sampling error. With only sampling error, the poll gives a so-called unbiased estimate of the population average, and so the errors should be uncorrelated with the actual outcome.
This doesn’t mean there was any election fraud. It just means that the exit poll estimates (above, I was using Edison/Mitofsky’s “best Geo estimator”; their “within-precinct error” gives very similar results) are not simply based on a random sample of all ballots cast in a precinct. As Edison/Mitofsky note on page 31 of their report, there are sources of error other than random sampling, most notably differental nonresponse. Perhaps these factors include votes taken during hours when the exit pollsters weren’t there or other coverage issues. In some elections, vote tallies are statistically stable over time, but it doesn’t have to be that way. Or maybe there were some other adjustments going on with the polls.
Summary
US Votes is correct to point out an inherent contradiction in the Edison/Mitofsky report, which is that it blamed the exit polls for the discrepancy while at the same time not seeming to look hard enough to find out where the problems were occurring. (To me, the most interesting table of the Edison/Mitofsky report came on page 35, where they report average within-precinct errors of 3.3%, 0.9%, 1.2%, 2.5%, and 1.1%–all in favor of the Democrats–in the five most recent elections. (Again, I’m dividing all their numbers by 2 to give errors in vote proportion rather than vote differential.))
The errors appear to be nationwide and would seem to be more consistent with nonresponse and undercoverage rather than something more local such as fraud.
Just scanning the web, I found more on this here, here, here, here, and here.
As Jeronimo said, let’s just hope this doesn’t happen in Mexico!
Full disclosure: Five years ago, I briefly consulted for Voter News Service and met Warren Mitofsky. I have no current conflicts of interest.
AG: Discrepancies are lowest with paper ballots and higher with election machines. I don't know that there are any reasonable hypotheses of fraud occurring with all machines (mechanical, touch screen, punch cards, optical scan).
DF: A big way that paper ballots differ from mechanical ballots is that there's more chance of operator error with machines.
If there were biases such that when a person accidentally voted for both candidates, the vote went to Bush (maybe just because he alphabetically came first), that could do it.
Yes, that's possible. But I'm pretty sure that operator errors wouldn't have such a large and consistent sign across all these states. I think nonresponse and undercoverage are more plausible explanations.
What gives with the negative slope (which is, just slightly, "statistically significant")?
This might be the most interesting result of all. One explanation is that people who voted for Bush felt motivated to lie about it. This motivation was strongest in states the were very pro-Kerry.
Even though I'd love for the data to show that there was tampering with the ballots, it doesn't really look that way. But it does look like the bias in the exit poll is more due to lying than sampling bias. This finding suggests that many people who voted for Bush were ashamed of doing do.
I would attribute the pattern to various individual differences among states that happen to correlate with vote for Kerry. The Edison/Mitofsky report gives the poll errors for each of the past 5 elections; as I recall, in all cases the polls overstated the Democrats' vote share.
Two comments:
1) The USCounts report to me actually seemed to have a bias that could have reflected a similar behavior to any potential sampling bias. I draw attention to their graph (page 10 of the pdf file) showing the vote discrepancy by recorded precint vote (quintiles of Bush votes). Such a graph may rule out the "reluctant Bush voter" – but could support the "emphatic Kerry voter" model.
I will get back to the replicated bias issue in a moment. Turning to the emphatic Kerry voter model, the idea is that Kerry voters in Bush strongholds would have higher than typical response rates; my anecdotal support of this theory came from direct observation of Kerry supporters (far more anti-Bush than pro-Kerry – and typically upset over the war). In other words, the strong emotional feelings of the anti-Bush vote resulted in an electorate that on one side had moderate feelings of keeping the status quo (Bush), while the other side passionately wanted him removed.
Back to the duality in behavior – the USCounts report seems to ignore the emphatic Kerry model altogether, and only seems to consider (and reject) the reluctant Bush model. Would this not be another example of an emphatic pro-Kerry bias? In other words, a reluctant Bush hypothesis is reasonable (because of passionate anti-Bush feelings), but USCounts writers are blinded enough by this viewpoint to fail to consider the other possibility.
2) (Quick comment) Widespread fraud allegations are not generally reasonable in the US in the information age. To pull off such a massive conspiracy, large numbers of people would have to be recruited, communicate, and operate discreetly. There would be way too many (economic and other) incentives for some participants to come "clean" and "leak" what happened. This is not credible claim. On the other hand, one cannot rule out local fraud, uncoordinated, happening on a large enough scale to cause some effects. This seems highly unlikely, however.
It would be nice to view the first two graphs with the x and y axes drawn. Then you could look at each one and make three assessments:
1. Are the X's systematically >0?
2. Are the Y's systematically >0?
3. Are X and Y correlated?
AG: The Edison/Mitofsky report gives the poll errors for each of the past 5 elections; as I recall, in all cases the polls overstated the Democrats' vote share.
DF: So you can ask whether the 2004 results were more biased or differently biased than the other years.
a. Is the magnitude of the presidential effect (6% off) comparable to the effect in 2000?
b. Did the Senate/President effect (less bias for Senate than for President) exist in 2000?
Greg,
Your point (1) sounds reasonable. I pretty much skipped over the details of the US Counts arguments on these points since I didn't really see where they were going. Your point (2) also makes sense, especially since the discrepancies were nationwide and not confined to one or two states.
Deb,
Most of the numbers you want are in the Edison/Mitofsky report, starting on page 21. The summary of previous years is on page 35.
I believe that if you do a "Rule of Nines" analysis of the county-by-county vote for president in Florida, and then compare that curve to the same analysis on another issue on the Nov. 2004 Florida ballot like the Florida constitutional amendment on minimum wages, you will see some very suspicious differences in deviation from the standard logarithmic Rule of Nines curve for the presidential vote, but not for the other issues.
Sorry to drop in on you folks, but I think the answer to the Senate/President question is fairly simple if you consider the fact that WPE is not a sound dependent measure for "survey error" or "bias."
See this analaysis by Elizabeth Little, former USCV contributor, which identifies the critical flaw in the USCV study:
http://www.geocities.com/lizzielid/WPEpaper.pdf
Ms. Liddle transformed WPE into a variable (bias index) that strips the confounding nature of precinct partisanship and leaves pure bias.
I know this is going to get jumbled, but try to follow the data from Survey USA. The last number to the right of each state is the "swing" between the margins Bush-Kerry (Pres race) and Rep-Dem (Senate race).
Pres Senate
State B-K Rep-Dem Swing
Colorado 3% -4% 7%
Arkansas 4% -6% 10%
Pennsylvania -1% 18% 19%
Washington -4% 6% 10%
California -11% -21% 10%
Florida 1% 0% 1%
Ohio 2% 28% 26%
Missouri 5% 19% 14%
North Carolina 8% 5% 3%
South Carolina 11% 11% 0%
Kentucky 21% 9% 12%
Oklahoma 30% 9% 21%
Georgia 12% 16% 4%
Indiana 19% -27% 46%
Illinois -12% -39% 27%
These data clearly suggest that there was a lot of ticket splitting going on in the 2004 elections.
Since the actual tickets were split as strongly suggested by the pre-election polling, it is not likely that the WPE for Senate would match the presidential WPE.
But even with Ms. Liddle's variable, I don't think it would be a useful measure for comparison. In states where ticket splitting was very common (I suggest all the states I list above except for Florida, the Carolinas, and Georgia), I suggest that there is too much noise to draw any real conclusions.
Assume that Kerry voters are oversampled at a ratio of 1.5:1. How can we know if oversampled Kerry voters split their tickets? How do we know if a Democrat who voted for Bush did not split her/his ticket? That is, "bias" is measured by oversampling of supporters of one candidate over another. Just because there was "bias" in the presidential exit poll, doesn't mean that there will also be bias in the Senate exit poll.
My point is that it seems reasonable that for a consistent and geographically dispersed level of Kerry bias in the presidential exit poll, there would not be an equally consistent and dispersed Democratic bias in Senate exit polls, even if WPE was a perfect measure of bias.
Rick,
What you say sound reasonable to me. As I noted in the blog entry, the states with more split-ticket voting (as crudely measured by the absolute difference between Democratic Senatorial and Presidential vote shares) do show bigger absolute differences between Senate and Presidential errors.
Lets remember that the polls leading us to the election consistently had Bush in the lead. So we might be able to bring that data to bear (and the polling methodologies) when analyzing the results.
Exit polling seems more personal to many people than telephone polls (a potential factor). And of course, there is the temporal dicontinuity factor as well (before and after the fact).
o.k, while it is good to see you aren't rushing to endorse USCountVotes, and your analysis shows they don't make a strong case, there are other factors to consider.
In particular, I would like to know what you have to say about the comments made here:
http://www.renewamerica.us/columns/wambough/05010…
If the exit-poll data itself were biased, and/or media-influenced turnout affected the final count, wouldn't that render an even statistically "correct" USCV study disengenuous, and undermine any subsequent conclusions based on assumptions of it's validiy?
In discussing the discrepancies between exit polls and final results, I would like you to address the fact that electronic voting is easily altered at the precinct level and at the final canvassing level in the county. With Excel my husband wrote a four step program to alter the individual votes and change everything so that there was no way to trace what had been done. In Lee county, Florida Black Box was easily able to show how you can manipulate final tallies from optical scanners. Furthermore all the complaints that I have found about vote switching on DRE machines across all the states have always shown a bias only for Republicans. No one reported that the DRE machine changed a Republican vote to a Democratic one. Furthermore you must remember that the voting machines are controlled only by the manufacturer; the individual county can not program their machines for an election. Finally there has been a tremendous reduction in percentage of nonvotes on DRE machines in the 2004 election for president. In 2004 DRE results showed about .7% non votes for a president. In the past elections the total was much higher, approximately 3.4%. It is wuite plausable that with a built in machine bias for Republicans, you can take those 3% of 'lost or non votes for a president and make them all for the Republican candidate. There are so many ways of altering DRE outcomes since there is no way to audit how the outcome relates to actual voter intent. Without an immediate random audit of voting machine tallies at the precinct level before an election result is reported, there is no way to ignore the dangers of black box voting. I would appreciate comments on the below listed aricle: History of the Debate Surrounding the 2004 Presidential Election http;//electionarchive.org/evAnalysis/US/Presidential-Election-2004.pdf