November 2006 Archives

One of my favorite points arose in the seminar today. A regression was fit to data from the 2000 Census and the 1990 Census, a question involving literacy of people born in the 1958-1963 period. The result was statistically significant for 2000 and zero for 1990, which didn't seem to make sense, since presumably these people were not changing much in literacy between the ages of 30 and 40. The resolution of this apparent puzzle was . . . the difference between the estimates was not itself statistically significant! (The estimate was something like 0.003 (s.e. 0.001) for 2000, and 0.000 (s.e. 0.002) for 1990. So both data points were consistent with an estimate of 0.002 (for example). But at first sight, it really looked like there was a problem.

P.S. These was a small effect, as can be seen by the fact that it was barely statistically significant, even though it came from the largest dataset one could imagine: the Chinese census. Well, just a 1% sample of the Chinese census, but still . . .

Maybe people in India aren't so happy as we thought. The British Psychological Society Research Digest points to this press release:

Adrian White . . . analysed data published by UNESCO, the CIA, the New Economics Foundation, the WHO, the Veenhoven Database, the Latinbarometer, the Afrobarometer, and the UNHDR, to create a global projection of subjective well-being: the first world map of happiness. . . . The meta-analysis is based on the findings of over 100 different studies around the world, which questioned 80,000 people worldwide. . . . It is worth remembering that the UK is doing relatively well in this area, coming 41st out of 178 nations. Further analysis showed that a nation's level of happiness was most closely associated with health levels (correlation of .62), followed by wealth (.52), and then provision of education (.51). The three predictor variables of health, wealth and education were also very closely associated with each other, illustrating the interdependence of these factors. There is a belief that capitalism leads to unhappy people. However, when people are asked if they are happy with their lives, people in countries with good healthcare, a higher GDP per captia, and access to education were much more likely to report being happy. We were surprised to see countries in Asia scoring so low, with China 82nd, Japan 90th and India 125th. These are countries that are thought as having a strong sense of collective identity which other researchers have associated with well-being. It is also notable that many of the largest countries in terms of population do quite badly. With China 82nd, India 125th and Russia 167th it is interesting to note that larger populations are not associated with happy countries.

My first thought upon reading this was amusement at the statement, "the UK is doing relatively well in this area, coming 41st out of 178 nations." They're so modest in the U.K.! Can you imagine someone in the U.S. being happy about being ranked 41st?

For a more scholarly take on all this, you can check out this article by Helliwell and Putnam and this article by Helliwell.

Missing-data imputation is to statistics as statistics is to research: a topic that seems specialized, technical, and boring--until you find yourself working on a practical problem, at which point it briefly becomes the only thing that matters, the make-it-or-break-it step needed to ensure some level of plausibility in your results.

Anyway, Grazia pointed me to this paper by a bunch of my friends (well, actually two of my friends and a bunch of their colleagues). I think it's the new state of the art, so if you're doing any missing-data imputation yourself, you might want to take a look and borrow some of their ideas. (Of course, I also recommend chapter 25 of our forthcoming book.)

Pretty funny:

A twice-fired cop who got her job back after being sacked in 1992 isn't entitled to a third stint with the NYPD, a state appeals court has ruled. Angela Willis - who was canned in 1995, only to be reinstated and fired again in 2004 - had her latest bid to keep her job unanimously rejected by a five-judge panel. The state Appellate Division ruling, which upheld her ouster by a Police Department trial, was made public yesterday. Willis had originally been dropped from the NYPD 11 years ago for sick leave abuse, racking up 40 absences. But she successfully sued to get reinstated. She then repeatedly ran afoul of her bosses during her second term in the NYPD. Police brass suspended her in 2000 for allegedly impeding the investigation of a Queens murder, in which her SUV was allegedly used as the getaway car. Six months later, she was charged with showing up to work drunk and with a forged doctor's note. She was suspended and police later moved to fire her. Willis sued the NYPD last year in federal court, charging that she had been fired for making harassment complaints. But the suit was withdrawn earlier this year.

But I still think this was funnier.

Thomas Trimbur sent the following question:

In a comment on this entry, Thom writes,

I'm not convinced that what we call happiness is a single thing. We could probably divide it into (at least) two concepts - local happiness "this instant" and general happiness. I think that having children relates more to the latter (or possibly towards a related concept like fulfilment).

Beyond that you'd need a theoretical account of happiness to make sense of what's going on. The (naive) economic analysis is that happiness leads to inaction, but the some theories of emotion propose the opposite (with evidence in support). For example the broaden and build theory of emotion proposes that the evolutionaty function of positive emotions is to build resources - so you'd maybe expect happy people to plan for the future (whereas we know very unhappy people don't).

I'm especially interested in his second comment--the point about action and inaction is something I'd never thought about. From an economic standpoint, if you are at a maximum of relative happiness, you would want to do what it takes to stay there (which might be inaction, but it might be to work your tail off, if, for example, you're happy but in major dept). For unhappy people, one could try a reverse explanation: if you're unhappy despite everything you've tried, then maybe giving up seems like the best alternative.

I ran across this paper by Michael Ross. Here's the abstract:

Many scholars claim that democracy improves the welfare of the poor. This article uses data on infant and child mortality to challenge this claim. Cross-national studies tend to exclude from their samples nondemocratic states that have performed well; this leads to the mistaken inference that nondemocracies have worse records than democracies. Once these and other flaws are corrected, democracy has little or no effect on infant and child mortality rates. Democracies spend more money on education and health than nondemocracies, but these benefits seem to accrue to middle- and upper-income groups.

This is an interesting idea. One of their key points is the datasets that are usually analyzed have missing-data patterns that bias the results. I am sympathetic toward this reasoning. Another issue is controlling for systematic differences between countries, so that the analysis is looking at countries that have transitions to and from democracy. I'm thinking that it might make sense to have two separate models for the two different transitions. Also, I'm wondering whether it would make sense to look at longer time lags. I have no quick solutions here, but it seems like it would be a good problem for a student to look at, to reanalyze the data and see what turns up.

On a more substantive direction, the last part of the paper has some discussion of why democracy might not be so great for the poor. But since the results are all comparisons with non-democracies, I'd think there should be some disucssion of the choices made by non-democratic regimes. (Or maybe this is there, and I'm just unfamiliar with this research area.)

Steven Levitt points to a report by Kate Holton comparing self-reported happiness levels in different countries. Holton wrote:

Young people in developing nations are at least twice as likely to feel happy about their lives than their richer counterparts, a survey says. Indians are the happiest overall and Japanese the most miserable. According to an MTV Networks International (MTVNI) global survey that covered more than 5,400 young people in 14 countries, only 43 percent of the world's 16- to 34-year-olds say they are happy with their lives. MTVNI said this figure was dragged down by young people in the developed world, including those in the United States and Britain where fewer than 30 percent of young people said they were happy with the way things were. . . . "The happier young people of the developing world are also the most religious," the survey said. The MTVNI survey took six months to complete and resulted in the Wellbeing Index which compared the feelings of young people, based on their perceptions of how they feel about safety, where they fit into society and how they see their future. Young people from Argentina and South Africa came joint top in the list of how happy they were at 75 percent. The overall Wellbeing Index was more mixed between rich and poor. India came top followed by Sweden and Brazil came last.. . . The 14 countries included in the survey were Argentina, Brazil, China, Denmark, France, Germany, India, Indonesia, Japan, Mexico, South Africa, Sweden, the UK and the U.S.

Levitt basically says he doesn't believe these results because, as he puts it,

Economists have a notion called “revealed preference.” By looking at people’s actions, you can infer how they feel. Applied to this MTV survey, if their measure of happiness or Wellbeing Index were meaningful, then I would expect that we would see a steady flow of unhappy young people from the United States and the United Kingdom immigrating to happy places like South Africa and Argentina and Wellbeing places like India.

History tells us that the flow of immigrants has always been and continues to be in the other direction, which to an economist, is the strongest evidence that whatever people are looking for, developed countries like the United States are where they are finding it.

I don't know the details of the survey; for example, maybe the pollsters have more difficulty reaching unhappy people in some countries than others. But let's assume that it's doing a good job of getting people's attitudes. Levitt writes that "people make many mistakes in forecasting what will or will not make them happy in the future," but I don't see why this invalidates survey responses about current happiness levels. If anything, it would suggest that emigrants to the U.S. are, possibly mistakenly, basing their decisions on estimates of future happiness (or maybe possibilities for their children). I would think that, from an economist's perspective, it would be completely reasonable for a currently-happy person in India (say) to come to the U.S. in anticipation of future happiness for self and family. Even if this anticipation turns out to be wrong, the decision to emigrate will be based on the feelings at that time, not on their future happiness levels.

Beyond this, I wonder if Levitt is falling into a "Simpson's paradox" trap of confusing within-group and between-group comparisons. The Indians who emigrate to America are not a random sample of Indians, and so it is possible for (a) Indians to be happier than Americans, and (b) Indian immigrants to become happier when moving to America. (But, given that people can make mistakes in forecasting their future happiness, I don't know that (b) is true.)

To put it another way, I don't plan to emigrate to India. But, even if the average Indian is happier than the average American, it doesn't mean that I'd be happier if I were to emigrate. It's the difference between correlation (the observed pattern of Indians and Americans) and causation (what would happen to an individual person if he or she were to move).

Anyway, I don't mean to belabor the point, it's just something I'd think an economist would be more aware of. Or, more likely, there's another twist to the argument that I'm missing (for example, some reasoning about equilibria).

In summary . . .

Michael Herron pointed me to this paper by Laurin Frisina, James Honaker, Jeffrey Lewis, and himself, "Ballot Formats, Touchscreens, and Undervotes: A Study of the 2006 Midterm Elections in Florida". Here's the abstract:

The 2006 midterm elections in Florida have focused attention on undervotes, ballots on which no vote is recorded on a particular contest. This interest was sparked by the high undervote rate—more than 18,000 total undervotes out of 240,000 ballots cast—in Florida’s 13th Congressional District race, a race that, as of this paper’s writing, was decided by 369 votes. Using precinct-level voting returns, we show that the high undervote rate in the 13th Congressional District race was almost certainly caused by the way that one county’s (Sarasota’s) electronic touchscreen voting machines placed the 13th Congressional District race above the Florida Governor election on a single screen. We buttress this claim by showing that extraordinarily high undervote rates were also observed in the Florida Attorney General race in Charlotte and Lee Counties, places where that race appeared below the Governor race on the same screen. Using a statistical imputation model to identify and allocate excess undervotes, we find that there is a roughly 90 percent chance that the much-discussed Sarasota undervotes were pivotal in the very close 13th Congressional District race. Greater study and attention should be paid to how alternatives are presented to voters when touchscreen voting machines are employed.

Tyler Cowen points to Robin Hanson who points to this paper by Olson, Vernon, Harris, and Jang, "The heritability of attitudes: a study of twins". Robin writes, summarizing the paper,

your differing attitudes on abortion, birth control, immigrants, gender roles, and race are mostly due to your genes, while your attitudes toward education, capitalism and punishment are due to your life experiences.

This interested me for two reasons:

1. The variation in political attitudes is inherently interesting--there is clearly a wide range of acceptable political beliefs in any society, and they can't simply be explained in terms of individual or group interests. And it's well known that your party ID is predictable from the party ID of your parents.

2. From a statistical perspective, I'm suspicious of sharp dividing lines such as in Robin's quote (A,B,C are mostly due to genes, X,Y,Z are mostly due to life experiences). In my experience, data don't usually separate things so clearly, but people can get confused by using statistical significance as an arbitrary criterion.

I'll give the abstract of the Olson et al. paper, then my thoughts. Here's the abstract:

I always tell students to give variables descriptive names, for example, define a variable "black" that equals 1 for African-Americans and 0 for others, rather than a variable called "race" where you can't remember how it's defined. The problem actually came up in a talk I went to a couple of days ago: a regression included a variable called "sex", and nobody (including the speaker) knew whether it was coding men or women.

P.S. Yet another example occurred a couple days later in a different talk (unfortunately I can't remember the details).

P.P.S. I corrected the coding mistake in the first version of the entry.

P.P.S. Check out Keith's story in the comments.

Jose von Roth writes,

The following appeared in an email list:

Alex Tabarrok links to this interview with Emily Oster, an economist who is studying ways of mitigating Aids in Africa. This is an area I know nothing about, but the following paragraphs caught my eye:

anthropologists, sociologists, and public-health officials . . . believe that cultural differences—differences in how entire groups of people think and act—account for broader social and regional trends. AIDS became a disaster in Africa, the thinking goes, because Africans didn't know how to deal with it.

Economists like me [Oster] don't trust that argument. We assume everyone is fundamentally alike; we believe circumstances, not culture, drive people's decisions, including decisions about sex and disease.

My quick comment on this is that everyone may be fundamentally alike, but apparently the culture of anthropology, etc., is associated with different attitudes than the culture of economics. (One could make a selection argument, of course, that people with attitudes like Oster's drift toward economics, and that people with the other attitudes drift toward anthropology, etc.--but that wouldn't fit with the assumption that "everyone is fundamentally alike." And I think it would be extreme and implausible cynicism to think that anthropoligists etc. and economists have different attitudes simply because of different incentive structures in their fields.)

A more measured response might be to say that political scientists accept that people sometimes have fundamentally different attitudes and interests (i.e., are not "fundamentally alike" in many social settings) and that social and political institutions can affect how they interact.

Circumstances and culture: are they like weather and climate?

This is not to say that I disagree with Oster on the substance of her argument. The key distinction, using her terminology, seems to be "circumstances" vs. "culture"--and at some level, "culture" is just a series of circumstances (or, conversely, the "circumstances" you see are affected by your culture). Political scientists would throw in the word "institutions" in here somewhere too, but it's the same general point.

Just to be clear: I'm not trying to be critical of Oster here--what I'm trying to do is understand the different attitudes in different social sciences, and the effects these have on research claims. In economics, as in many fields, I think that having a strong methodological preference can be helpful in focusing one's research. (That's the attitude I've always taken about Bayesian methods: if you work hard at constructing a good model, and you check it against data, you can learn a lot. But it helps to have that commitment to pushing the Bayesian approach hard and being willing to work with it.) Similarly, I expect that Oster's strong assumptions about individual behavior and strong affiliation with "economism" (an analogy to "Bayeisanism"?) can help her make progress by clarifying her thinking and giving her the fortitude to work out the full implications of her ideas.

How does this play out in practice? You can take a look at the following two articles:

On Explaining Asia’s “Missing Women”: Comment on Das Gupta by Emily Oster. Paper here.

Cultural versus Biological Factors in Explaining Asia's "Missing Women" by Monica Das Gupta. Paper here.

I have not tried to evaluate the competing arguments (hey, I'm busy too!), just to give these as a possible example of different approaches taken by economists and public-health researchers. I think the quote about people being fundamentally alike doesn't really come into play here, but perhaps these papers do illustrate different ways of studying a social phenomenon.

Greg Huber spoke on this last week. Here's the paper, and here's the abstract:

Do presidential campaign advertisements mobilize, inform, or persuade citizens? To answer this question we exploit a natural experiment, the accidental treatment of some individuals living in non-battleground states during the 2000 presidential election to high levels or one-sided barrages of campaign advertisements simply because they lived in a media market adjoining a competitive state. . . . In contrast to previous research, we find little evidence that citizens are mobilized by or learn from presidential advertisements, but strong evidence that they are persuaded by them. . . . Our research suggests that political advertising functions, in part, like propaganda, rather than as a purely benign source of information.

I don't have anything to add right now on the substance of this paper: the methods seem reasonable and the result is plausible. My only comment is about the discussion after the talk: several people (incluidng me) asked why it was considered surprising that advertising persuades. Huber responded that, in the specialized academic literature on political advertising, there has been a consensus in favor of the view (false, in Huber's estimation) that political advertising mobilizes but does not persuade. This seems like one of these cases where the less sophisticated view is more correct than the subfield experts'. Why can this be? For one thing, the experts may have too much of an investment in their particular theoretical framework.

I have noticed something similar in the voting power literature, where the experts in this subfield basically have it all wrong (for extended discussion of this point, see our BJPS article). I really have no idea how to communicate with the "voting power" people: their whole set of assumptions is so flawed, but I feel that they can't really hear what I'm saying (even if they were to listen; another difficulty, as far as I can tell, is getting people to notice). But political scientists outside of this subfield don't really take voting power very seriously (that's an appropriate attitude, actually).

Also, a couple of little comments on the Huber and Arceneaux paper:

D pointed me to this great multilevel modeling example:

By living in a well-to-do neighborhood, poor people increase their risk of death, according to a new study by School of Medicine researchers to be published in the December issue of the American Journal of Public Health. . . researchers found that death rates were highest among people of low socioeconomic status who also lived in affluent neighborhoods. That finding surprised the researchers, but "every way we looked at the data, we found the same result" . . .

Previous studies have shown that neighborhood plays an important role in an individual's health. Most studies have found that people fare better in high-income neighborhoods. The Stanford study is unique because it combined individual economic status with neighborhood status to produce a more detailed picture of the issue.

The researchers discovered the trend by analyzing data from another study that looked at the incidence of heart disease risk factors in California between 1979 and 1990. The study followed 8,200 men and women from 82 neighborhoods in Monterey, Modesto, Salinas and San Luis Obispo over 17 years.

The researchers used income and education to determine individual socioeconomic status. They then divided people into low, moderate or high socioeconomic groups. Similarly, the researchers classified neighborhoods as being of low, moderate or high socioeconomic status.

The study found that over time, the differences in death rates among the groups became more pronounced. After 17 years, 19 out of every 1,000 women of low socioeconomic status who lived in wealthier neighborhoods had died, compared with 11 of every 1,000 from poorer neighborhoods. The trend was similar, but less dramatic in men.

They found that age as well as a number of risk factors, such as obesity, hypertension and smoking, did not account for their results. There were also no differences in the causes of death, which were largely due to chronic diseases, the researchers said. They also found that access to neighborhood goods and services, such as health care, grocery stores, parks and gyms, did not explain their findings. . . .

The funny thing is that the graphs in the news article don't fully tell the story. They should plot death rates vs. socioeconomic status, with three separate lines on the graph, one for each neighborhood. They can then use different-sized circles to show the number of people in each category (i.e., the "superplot," as we did here).
(I also assume they controlled for age, although I didn't see that mentioned in the news article.)

Jean-Luc points me to Mysterious 'Neural Noise' Actually Primes Brain for Peak Performance:

In the November issue of Nature Neuroscience, the Rochester study shows that the brain's cortex uses seemingly chaotic, or "noisy," signals to represent the ambiguities of the real world—and that this noise dramatically enhances the brain's processing, enabling us to make decisions in an uncertain world.

"The cortex appears wired at its foundation to run Bayesian computations as efficiently as can be possible," says Pouget. His paper says the uncertainty of the real world is represented by this noise, and the noise itself is in a format that reduces the resources needed to compute it. Anyone familiar with log tables and slide rules knows that while multiplying large numbers is difficult, adding them with log tables is relatively undemanding.

The article cited here ("Young children aged between two and four years believe that you only have to hide your head to become invisible – if your legs are on view, it doesn’t matter, you still can’t be seen") is cool (and so unsurprising that I'm a bit surprised that it's news), and it's certainly psychology, but why is it characterized as "behavioral economics"? It seems cognitive (not behavioral), and I don't see the connection to economics at all!

Economists sure are competitive (at least, based on this sample of size 1).

Median often feels like an ad hoc calculation, not like an aspect of a statistical model. But in fact, median actually corresponds to a model. Last week, Risi and Pannagadatta at the Columbia machine learning journal club reminded me that L1 norm of the data is minimized at the point of the median. But the L1 norm and Laplace distribution are closely related (to be elaborated later), and the median effectively corresponds to the mu parameter of the Laplace distribution.

For example, let's assume a very flat prior on the Laplace parameters and data [1,2,10]. The posterior distribution of mu, as obtained through WinBUGS is shown in the histogram below. The MAP peak is at about 2, and we can see that it's hard to estimate the median with so little data:

I am sure that someone has thought of this before, but it's easier to reinvent it than to track it down. Let me now elaborate on the connection between norms and distributions.

Here.

Apparently people now make Lego sculptures with moving parts:

My favorite quote is about the desk:

The most interesting thing about this project was not the design of it . . . No, the part that took the most thought was the economy of pieces. While this desk was obviously costing the company a pretty penny, they still had a budget to consider, so I had to design the color scheme such that the bricks were used in a proportion which matched the Blue Tub distribution as closely as possible.

There's an interesting thread on the 2006 elections at Ask E.T., starting on May 9 and continuing to after the election. Also, unrelated to the election, the thread contains this pretty picture (cited in the Gerber and Malhotra paper):

Tyler Cowen links to an article by Jan Brueckner and Ann Largey that finds that people in suburbs have more social interaction than people in cities. Here's the abstract:

Various authors, most notably Putnam (2000), have argued that low-density living reduces social capital and thus social interaction, and this argument has been used to buttress criticisms of urban sprawl. If low densities in fact reduce social interaction, then an externality arises, validating Putnam’s critique. In choosing their own lot sizes, consumers would fail to consider the loss of interaction benefits for their neighbors when lot size is increased. Lot sizes would then be inefficiently large, and cities excessively spread out. The paper tests the premise of this argument (the existence of a positive link between interaction and density) using data from the Social Capital Benchmark Survey. In the empirical work, social interaction measures for individual survey respondents are regressed on census-tract density and a host of household characteristics, using an instrumental-variable approach to control for the potential endogeneity of density.

I'll have to read through this more carefully. To start with, I'd like to see some simple comparisons and scatterplots (average # friends of people who live in rural area, towns, suburbs, and cities). I'm sure there's a good reason for the regressions in the article, but, as always, I'd understand and trust results such as their Table 3 much better if I could first see some data patterns. It looks like the data are publicly available, so it shouldn't be a problem to do these analyses.

We're actually planning to do similar analyses with the data we've collected using the General Social Survey. One of our conjectures is that people living in cities have more acquantainces but fewer people they trust, compared to people who live in smaller communites. I'd always thought this would be consistent with Putnam's work on cohesive communities.

Dragomir Radev (who's visiting Columbia this year from the University of Michigan) is teaching this interesting-looking course this spring on search engine technology. He said that one of the assignments will be to build your own search engine, and another might be to build that machine that can figure out what you're typing just by listening to the sound of your keystrokes. It looks pretty cool--it's too bad it's offered at 6pm so I can't make it.

On a barely related note: the class meets once per week for two hours. I know that people often find this sort of once-a-week schedule convenient, but it's been my impression that research shows that people learn better from more frequently-scheduled classes. I usually do twice a week. I'd prefer meeting three times per week, but Columbia doesn't seem to do much of that anymore. According to Dave Krantz, when he was a student at Yale way back when, some of the courses would be Mon Wed Fri, and others would be Tues Thurs Sat. That's a bit too rigorous for my taste, but I wouldn't mind teaching some Mon Wed Fri courses.

P.S. Drago clarifies:

I [Drago] didn't say that I would give an assignment about recognizing what is typed. I said that this could be a nice course project.

The actual assignments that I am planning to give will be chosen from this list.

- build a search engine
- build a spam recognition system
- perform a text mining analysis of a large data set such as the Enron mail corpus or the Netflix movie recommendation data set
- perform an analysis of a large networked data set such as a snapshot of the web graph (e.g., write code to compute pagerank).

I would imagine that each student can only do 2 or 3 of them.

Then each student will also have to propose an independent course project where the sky is the limit. Some ideas include cross-lingual information retrieval, text summarization, question answering, identifying online communities, information propagation in blogs, etc.

Somebody asked us about our "red-state, blue-state" results (the pattern that we found in the 2000 and 2004 elections, in which income and Republican voting are highly correlated in poor states such as Mississippi, moderately correlated in medium states such as Ohio, and not at all correlated in rich states such as Connecticut--hence the title of our paper, "What's the Matter with Connecticut"; also see pretty picture here), and whether anything similar happened in 2006.

We're still looking for raw polling data; the closest we could find were these House exit poll results from CNN.com, which are broken down by income and region. Here are the results:

It's tougher to see this since we don't have the individual states, but income is indeed more correlated with Republican vote in the South and Midwest, and less so on the coasts, which fits our "red state, blue state" story.. (The income categories from the poll are 0-15,000, 15-30,000, 30-50,000, 50-75,000, 75-100,000, 100-150,000, and 150-200,000, and 200,000+; for sample size reasons we combined the two highest categories.)

John Kastellec checked the election results and found the Democrats to have received 54.8% of the average district vote, not 56%, as I had posted a couple of days ago. (My mistaken calculation of 56% was based on combining two different data sources which apparently counted third-party votes differently.)

The other thing we discussed in that previous entry was the distinction between average district vote and total vote share. The Democrats won in districts with, on average, lower turnout:

It turns out that the actual election outcome fit our predicted seats-votes curve:

Finally, John made some histograms of vote shares:

Lots of close races, it seems.

This is actually the curve John Kastellec estimated for 2006 using the 2004 election data (it's in our paper):

But the curve as estimated from the 2006 elections (once we have all the data in a convenient place) will look similar. It's basically consistent with what happened (with 56% of the vote, the Democrats got a bit more than 53% of the seats).

The historical pattern of votes is shown here. In 2006, the Democrats matched their historical performance in the 1960s-1980s in votes, but not in seats.

Our 1991 paper has more background on historical seats-votes curve.

Update: The numbers changed a bit since this entry was posted the other day. The Democrats got 55% of the average district vote, not 56%. (The confusion came because we used numbers from the New York Times that counted third-party votes in a different way than we did, I think; see note at the end of this entry.)

Back to the main story:
The Democrats' victory in the 2006 election has been compared to the Republicans' in 2004. But the Democrats actually did a lot better in terms of the vote. The Democrats received 54.8% of the average district vote for the two parties in 2006, whereas the Republicans only averaged 51.6% in 1994.

There was a big jump in 2006. Here's the time series:

(Bigger version is here.) The shaded areas on the graph show the periods where Republicans have controlled the House. The 2006 outcome of 55% for the Democrats is comparable to their typical vote shares as the matjority party in the decades preceding the 1994 realignment.

54.8% of the vote, 53.3% of the seats

Even with their large vote majority, the Democrats only received 53.3% of the seats in the House. This is as we and Bafumi et al. anticipated. More info on the seats-votes relationship is in our recent paper. (For example, had the Republicans received 54.8% of the vote in 2006, we estimate they would've won about 245 seats.)

By the way, the Democrats' 54.8% share of the two-party vote tracks closely with the "generic congressional vote" in which they were getting 56% in the polls (that is, 52.1%/(52.1% + 40.6%)).

Technical notes

We actually calculate average district votes by imputing 75% for uncontested races (to represent the strength that the party might have had if the district had been contested; the 75% comes from computing the average vote in districts just before and just after being uncontested, based on historical data), so we needed to make corrections for uncontesteds. In 2004, we have 31 uncontested Democrats and 37 uncontested Republicans: the average district vote for the Democrats was 50.5% using our correction (or 50.0% if you simply plug in 100% for uncontested races). In 2006, there were 45 uncontested Democrats and 10 uncontested Republicans, yielding an average district vote of 54.8% using our correction (or 56.8% if you simply plug in 100%). The 56.8% number is more dramatic but I think it overstates the Democrats' strength in giving them 100% in all those districts.

Another option is to use total vote (see here) rather than average district vote. We discuss this in Section 3.3 of our paper (in particular, see Figure 4). The short answer is that we use average district vote because it represents total support for the parties across the country. The Democrats tend to do better in lower-turnout districts and so their total vote is typically slightly lower than their average district vote. See the scatterplot here.

From Michael Finnegan in the LA Times, with a straight face:

The margin (Poizner beat the Democrat, 51% to 39%, with more than 10% favoring other candidates) suggests that Bustamante's TV ads highlighting the rotund lieutenant governor's weight loss failed to build voter confidence in his qualifications to regulate insurers.

Adam Sacarny noticed some dice in my office today and we came up with a good idea for a web-accessible random number generator: Put a die in a clear plastic box attached to something that can shake it. Train a video camera on the box, and pipe it to some digit-recognition software. Then whenever someone clicks on a button on the website, it shakes the box and read off the number. (We can use one of those 20-sided dice with each digit written twice, so we get a random number between 0 and 9.) Pretty convienent, huh?

I'd like to set this up but I assume somebody's done it already. In any case it's not nearly as cool as that program that figures out what you're typing by listening to the sounds of the keystrokes.

Jeronimo Cortina, Rodolfo de la Garza, and Pablo Pinto find, surprisingly, that the ability to speak both English and Spanish has a surprisingly small association with income among Hispanics in the U.S., with the association actually being negative for managerial jobs. They write,

These findings are troubling for several reasons. They suggest that the difference in earnings may be the consequence of discrimination in labor markets. Alternatively, it is plausible that lower wages may reflect the extent to which Spanish-speaking Latinos including those who are fluent in English, receive educational services of lower quality than Hispanics that speak English only only, and even non-Hispanic whites despite similar education attainment levels.

From a statistical perspective, this sort of analysis is interesting because it is of the "dog that didn't bark" variety: not finding an expected effect, which implies that there must be something cancelling the underlying pattern (of better skills--in this case, bilinguilism--yielding higher incomes). The regressions control for a bunch of variables (education, sex, age, citizenship, region, and occupation category). I wouldn't mind seeing an analysis using matching as well. It's a challenging problem to think of causally, since the point is that they're not simply estimating the causal estimate of biligual ability--they're actually trying to demonstrate that the model has a omitted variables.

And, of course, ...

I wanted to post the above picture because this is advice I'm giving all the time. The asterisk on the lower right represents the scenario in which problems arise when trying to fit the desired complex model. The dots on the upper left represent successes at fitting various simple versions, and the dots on the lower right represent failures at fitting various simplifications of the full model. The dotted line represents the idea that the problems can be identified somewhere between the simple models that fit and the complex models that don't.

Typically, we start at the upper left (with simple models we understand) and the lower right (the models we're trying to fit), and we debug by moving from both ends toward the middle to find where things break down.

(from chapter 19 of the new book)

In anticipation of tomorrow's election, I'd like to repost this entry from 2004 explaining why it's rational to vote. I was talking there about the Presidential election but the argument is relevant for Congress also (that is, if you, unlike me, would have the chance to vote in any closely contested elections):

The chance that your vote will be nationally decisive is, at best, about 1 in 10 million. So why vote?

Schematic cost-benefit analysis

To express formally the decision of whether to vote:

U = p*B - C, where

U = the relative utility of going and casting a vote
p = probability that, by voting, you will change the election outcome
B = the benefit you would feel from your candidate winning (compared to the other candidate winning)
C = the net cost of voting

The trouble is, if p is 1 in 10 million, then for any reasonable value of B, the product p*B is essentially zero (for example, even if B is as high as $10000, p*B is 1/10 of one cent), and this gives no reason to vote.

The usual explanation

Actually, though, about half the people vote. The simplest utility-theory explanation is that the net cost C is negative for these people--that is, the joy of voting (or the satisfying feeling of performing a civic duty) outweighs the cost in time of going out of your way to cast a vote.

The "civic duty" rationale for voting fails to explain why voter turnout is higher in close elections and in important elections, and it fails to explain why citizens give small-dollar campaign contributions to national candidates. If you give the Republicans or Democrats $25, it's not because you're expecting a favor in return, it's because you want to increase your guy's chance of winning the election. Similarly, the argument of "it's important to vote, because your vote might make a difference" ultimately comes down to that number p, the probability that your vote will, in fact, be decisive.

Our preferred explanation

We understand voting as a rational act, given that a voter is voting to benefit not just himself or herself, but also the country (or the world) at large. (This "social" motivation is in fact consistent with opinion polls, which find, for example, that voting decisions are better predicted by views on the economy as a whole than by personal financial situations.)

In the equation above, B represents my gain in utility by having my preferred candidate win. If I think that the Republicans (or the Democrats) will benefit the country as a whole, then my view of the total benefit from that candidate winning is some huge number, proportional to the population of the U.S. To put it (crudely) in monetary terms, if my candidate's winning is equivalent to an average $100 for each person (not so unreasonable given the stakes in the election), then B is about $30 billion. Even if I discount that by a factor of 100 (on the theory that I care less about others than myself), we're still talking $300 million, which when multiplied by p=1/(10 million) is a reasonable $30.

Some empirical evidence

As noted above, voter turnout is higher in close elections and important elections. These findings are consistent with the idea that it makes more sense to vote when your vote is more likely to make a difference, and when the outcome is more important.

As we go from local, to state, to national elections, the size of the electorate increases, and thus the probability decreases of your vote being decisive, but voter turnout does not decrease. This makes sense in our explanation because national elections affect more people, thus the potential benefit B is multiplied by a larger number, canceling out the corresponding decrease in the probability p.

People often vote strategically when they can (in multicandidate races, not wanting to "waste" their votes on candidates who don't seem to have a chance of winning). Not everyone votes strategically, but the fact that many people do is evidence that they are voting to make a difference, not just to scratch an itch or satisfy a civic duty.

As noted above, people actually say they are voting for social reasons. For example, in the 2001 British Election Study, only 25% of respondents thought of political activity as a good way to get "benefits for me and my family" whereas 66% thought it a good way to obtain "benefits for groups that people care about like pensioners and the disabled."

Implications for voting

First, it can be rational to vote with the goal of making a difference in the election outcome (not simply because you enjoy the act of voting or would feel bad if you didn't vote). If you choose not to vote, you are giving up this small but nonzero chance to make a huge difference.

Second, if you do vote, it is rational to prefer the candidate who will help the country as a whole. Rationality, in this case, is distinct from selfishness.

See here for the full paper (joint work with Aaron Edlin and Noah Kaplan) to appear in the journal, Rationality and Society.

This (sent to me several months ago by Will Fitzgerald) is so great I had to run it again:

Here's the background on it.

Adam Berinsky is presentjng this paper at the New York Area Political Psychology Meeting today. I don't have much to say about the content of the paper, except that a key issue would seem to me to be framing: are civil liberties a luxury (as our math professors would say in college when proving a theorem, "culture") that we can't afford in wartime, or are civil liberties a form of security that is needed more than ever during a war? I would think that many of the controversies about civil liberties--in policy discussions and in public opinion--depend on this framing.

In any case, I have some comments about the graphs in the paper. First, I like how the paper follows in the Page and Shapiro tradition of presenting results graphically rather than as tables. For the Berinsky paper, I'd recommend more consistency in the presentation, basically displaying the information, wherever possible, as line plots with time on the x-axis. This parallelism will make the paper easier to read, I think--partly because the graphs can be made physically small and thus fit into the text better, also because a compact display allows more information to be displayed and be made visible in one place (so that the reader--and the researcher--can see more comparisons and learn more).

In detail:

Steve Wang writes,

Boris sent this in:

Andy Stern, president of the Service Employees International Union--and himself an Ivy League graduate--recently said that the perception of Democrats as "Volvo-driving, latte-drinking, Chardonnay-sipping, Northeast, Harvard- and Yale-educated liberals" isn't a perception at all, but rather "the reality. That is who people see as leading the Democratic Party. There's no authenticity; they don't look like them. People are not voting against their interests; they're looking for someone to represent their interests."

In this case, the reporter (Thomas Edsall) is not making the mistake (as did Michael Barone and others earlier) but rather is reporting others' take on the issue. Actually introduces a different point, which is Dem (or Rep) leaders, as compared to Dem (or Rep) party members. No doubt that the leaders of both parties (as of just about all organizations) are richer than the general membership. It's considered to be more of a problem for the Dems, however, possibly because the Democrats are supposed to be "the party of the people." The fact that the Republicans are led by "Benz-driving, golf-playing, Texas, Harvard- and Yale-educated conservatives" is not such a problem because, in some sense, the Republicans never really claim to be in favor of complete equality.

This entry by Will Wilkinson reminded me of something that's bugged me for awhile, which is the use of term "loss aversion" to describe something that I'd rather call "uncertainty aversion," if that. (Wilkinson doesn't actually do this thing that irritates me--he actually is talking about loss aversion, referring to actual aversion to loss--but he reminds me of this issue.)

As I wrote before,

If a person is indifferent between [x+$10] and [55% chance of x+$20, 45% chance of x], for any x, then this attitude cannot reasonably be explained by expected utility maximization. The required utility function for money would curve so sharply as to be nonsensical (for example, U($2000)-U($1000) would have to be less than U($1000)-U($950)). This result is shown in a specific case as a classroom demonstration in Section 5 of a paper of mine in the American Statistician in 1998 and, more generally, as a mathematical theorem in a paper by my old economics classmate Matthew Rabin in Econometrica in 2000. . . .

Matt attributes the risk-averse attitude at small scales to "loss aversion." As Deb points out, this can't be the explanation, since if the attitude is set up as "being indifferent between [x+$10] and [55% chance of x+$20, 45% chance of x]", then no losses are involved. I attributed the attitude to "uncertainty aversion," which has the virtue of being logically possible in this example, but which, thinking about it now, I don't really believe.

Right now, I'm inclined to attribute small-stakes risk aversion to some sort of rule-following. For example, it makes sense to be risk averse for large stakes, and a natural generalization is to continue that risk aversion for payoffs in the $10, $20, $30 range. Basically, a "heuristic" or a simple rule giving us the ability to answer this sort of preference question.

There was some discussion of this on the blog last year. To recap briefly, no, I don't think this example is loss aversion, since no losses are involved. Yes, you could shift the problem by subtracting, to get losses, but that's not how it's framed. Getting back to the $40,$50,$60 example: if you want, you can say that the very mention of the $50 makes anything less seems like a loss, but I don't see it. I think the evidence is that people react to actual losses much more strongly than to a non-gain.

Risk aversion. No, it's loss aversion. No, it's uncertainty aversion. No, it's rule-following.

Anyway, my problem here is with "loss aversion" used in an automatic way to summarize various aspects of irrationality (such as avoidance of expected monetary value for small dollar amounts). My take on it (which is probably historically inaccurate) was that decision scientists first simply assumed that people used expected monetary value. Then they coined the term "risk aversion" and associated it with concave utility functions. Simple calculations (such as mine and Matt's, mentioned above) made it clear to many people (eventually everyone, I hope) that the typical non-EMV attitudes cannot be sensibly fit into an expected-utility framework. This led to ideas such as prospect theory which had aspects of expected utility but with biases caused by framing, confusions about probability, loss aversion, and so forth.

Now loss aversion is the catchphrase--and I agree, it's an improvement on the now-meaningless "risk aversion"--but I think it's silly to apply "loss aversion" to settings with no losses. Really, in some of these settings, I don't see "aversion" at all but rather a preference for certainty (perhaps "uncertaintly aversion") or even just the following of a rule.

The big issue

The big issue pointed out implicitly by Wilkinson (and others) is that people often seem to respond to the trend rather than the absolute level of the economy. I'm certainly not meaning to imply that, in battling over terminology, I'm resolving these deeper issues. My goal here is simply to point out that some commonly-used terms can have misleading implications.

Regarding Wilkinson's actual entry, his discussion is interesting, but I'm confused by his main point, which seems to be:
(a) Middle-class Americans shouldn't be so scared about losses--they'd still be able to get by OK on half their incomes.
(b) By being less afraid of losses, a middle-class American could take more risks which could result in a doubling of his or her income.
But, if point (a) is true, and you could easily live on half, then what's the motivation to double your income? Shouldn't we all just be taking more vacations?

I'm not trying to disagree with Wilkinson's point that many people's economic lives might not be so precarious as they think--as he puts it, middle-class Americans get a lot of things for free. I just don't see why this implies that people should be taking more risks.