I dodged a bullet the other day, blogorifically speaking. This is a (moderately) long story but there’s a payoff at the end for those of you who are interested in forecasting or understanding voting and public opinion at the state level.
Act 1
It started when Jeff Lax made this comment on his recent blog entry:
Nebraska Is All That Counts for a Party-Bucking Nelson
Dem Senator On Blowback From His Opposition To Kagan: ‘Are They From Nebraska? Then I Don’t Care’
Fine, but 62% of Nebraskans with an opinion favor confirmation… 91% of Democrats, 39% of Republicans, and 61% of Independents. So I guess he only cares about Republican Nebraskans…
I conferred with Jeff and then wrote the following entry for fivethirtyeight.com. There was a backlog of posts at 538 at the time, so I set it on delay to appear the following morning.
Here’s my post (which I ended up deleting before it ever appeared):
Party-Bucking Nelson May Be Nebraska-Bucking as Well
Under the headline, “Nebraska Is All That Counts for a Party-Bucking Nelson,” John Stanton writes:
Sen. Ben Nelson is catching a lot of grief from within his party for being the first Democratic Senator in decades to oppose his president’s pick for the Supreme Court. But the Nebraskan is hearing none of it.
“Are they from Nebraska? Then I don’t care,” a defiant Nelson said Tuesday.
What do people from Nebraska think about the Kagan nomination? Applying multilevel regression and poststratification (“Mister P”) to recent ABC poll data, Jeff Lax, Justin Phillips, and John Kastellec estimate that 62% of Nebraskans with an opinion favor confirmation… 91% of Democrats, 61% of Independents, and 39% of Republicans in the state. As Jeff puts it, it seems that Nelson only cares about Republican Nebraskans…
To step back a bit, where do these numbers come from? A national poll of 1500 Americans will include about 10 people from Nebraska, so these estimates of 91%, 61%, 39% can’t be coming directly from the data. What the multilevel model is doing is estimating how public opinion varies by demographic factors (sex, ethnicity, age, education), party identification, and state-level ideology. So the estimate for the Nebraska groups is based on what the survey finds for these partisan groups (and, to a lesser extent, on demographic breakdowns within Nebraska) and on Nebraska’s status (based on previous surveys) as a relatively conservative state.
It’s possible that actual conditions in Nebraska are quite a bit different. And I’m sure that Ben Nelson knows a lot of things about his state’s public opinion that aren’t included in our model. But, absent any other information, I’d go with Jeff, Justin, and John’s estimates and say that Nebraskans are pretty strongly in favor of Kagan’s nomination.
That looks good, right? So why did I spike it? Because more information came in. (As we say in statistics, a bit of falisification.) Here’s the story . . .
Act 2
Luckily, Jeff and I exchanged more emails, and I realized that I wasn’t aware of the whole story. Jeff pointed me to this link, which featured Sen. Nelson’s discussion of Nebraska opinion. I tried to patch things by adding the following to my pending bolg at fivethirtyeight.com:
P.S. Jeff points out, in Nelson’s defense, that one thing their poll analysis doesn’t capture is intensity. Nelson said, “there’s a constituency not to vote for her. There’s not a strong constituency to vote for her.”
P.P.S. Brian Beutler also reports that “Last month, a Republican poll found Kagan had only 33 percent support in Nebraska, with 48 opposed to her confirmation.” I don’t know what to think about that one. According to the link, the poll found Kagan’s nomination having only 41% support in Wisconsin (with 43% opposed) and with 28% support in Arkansas (with 54% opposed). That’s certainly possible–as noted above, the ABC poll that my colleagues analyzed has small sample sizes in any given state–but it’s not so plausible to me that these three states would differ so much from the model’s prediction based on national data. But I don’t really know.
So far, so good. Ass covered, blog ready to roll. Right? Not so fast.
Act 3
Jeff and I still weren’t sure what was going on. The Mister P estimates had Kagan with well over 50% support (among those with a preference) in nearly every state, which seemed to contradict not just the “Republican poll” mentioned above but everything by Gallup as well–they’ve polled on the issue more than once.
Were we sure about that 62% in Nebraska–sure enough for fivethirtyeight.com? I looked up Gallup and they had her at 46% in favor, 36% against, and 18% no opinion. That’s 46/(46+36) = 56% in our language. If so, we weren’t so sure about Nebraska at 62%.
That’s Gallup = 56% for the whole U.S. ABC = 62% for Nebraska, thus something like 68% for the whole U.S. 56% vs 69% is a big difference. Also, if they do differ, I’m inclined to believe the Gallup since it’s consistent with earlier polls. So then Nebraska’s probably around 50%. This doesn’t destroy my point above, but it changes things a bit.
Jeff clarified for me:
Sure, the ABC poll is out of whack with the newer Gallup poll, but for the ABC poll data there is no reason to think the estimates are off. If we have to scrap this point, it’s bc the polls disagree, not bc of MRP doing anything wrong.
After another email exchange, I concluded that,
With Nebraskans estimated to be split 50-50 and Republicans more likely to turn out to vote, and Nelson desperate to find a way to buck the party while not voting No on something popular . . . it’s not really a surprise at all what he’s doing. So I’ll have to scrap the 538 post.
Act 4
But what’s the larger story, if any? Is this like one of those Brady Bunch episodes where Bobby does something really stupid and then, when everything is resolved at the end of the half hour, and Mike and Carol ask what lesson he learns, he responds with way too specific, so that it’s clear that he didn’t really learn the deeper lesson?
The shallow lesson is certainly to be really careful about posting anything on fivethirtyeight.com or making off the cuff blog comments about Nebraskans.
But there’s a deeper lesson too, a lesson about how to understand and interpret state-level polls and election results.
Opinion swings can be decomposed into components: national swing, regional swings (relative to U.S.), state swings (relative to regions), democraphic swings (relative to the whole), etc. It’s the usual model of main effects + interactions. In situations such as the Kagan vote, we have lots of sources on national opinion, not so much on the rest. A decomposition allows you to estimate national swing more precisely. If our only goal were to summarize “the ABC poll data,” then you’re fine. But that’s not your goal. But our real goal is to estimate opinion, and for that you can’t hide behind ABC. (Not that there’s anything wrong with the ABC polls; to estimate national opinion we’d use ABC, Gallup, and everything else we can get our hands on. The point is that national averages are much more accessible than the individual-level data that we need to do Mister P. We can use national polls for national opinion and use Mister P to estimate relative opinion.)
The message for Mister P is that estimates should be adjusted using available national information. We did this in our election maps (adjusting to state-by-state vote) and also in our health care maps (by plotting opinion relative to 50%), and it’s in my (not quite finished) paper with Yair and in the mrp package, I think. Kari Lock and also discuss the point more generally in our recent paper to appear in Political Analysis.
Reprise
When forecasting or interpreting state-level votes or opinion swings, decompose into two parts: national swing, and state swings relative to each other. For example, if, like Jeff, Justin, and John, you do a state-level analysis and estimate that Nebraska is 7 percentage points less Kagan-supporting than the country as a whole, then report that difference. And if you want to estimate Nebraska itself, estimate national opinion as accurately as possible–that’s not difficult to do, it’s easy to get summaries from national polls–and then shift over to estimate Nebraska. (So if your best estimate of the national level of support were 55%, say, then you’d subtract 7 points and get an estimate of 48% for Nebraska, given all the information available to you.) Similarly for subgroups within states such as Democrats/Independents/Republicans, likely voters, young people, whatever.
In our paper, Kari and I discuss how to apply this approach to forecasting presidential elections by state, but the idea is really more general than that, and is something we should all be aware of when reporting state polls or state-level estimates from national polls.