Results matching “R”

As I've discussed here on occasion, I like to standardize continuous regression inputs by dividing by two standard deviations. That way the rescaled variables each have sd of 1/2, which is approximately the same sd as any binary predictor, allowing the coefficients to be interpreted together.

Standardizing is often thought of as a stupid sort of low-rent statistical technique, beneath the attention of "real" statisticians and econometricians, but I actually like it, and I think this 2 sd thing is pretty cool.

As Aleks pointed out, however, standardizing based on the data is not strictly Bayesian, because the interpretation of the model parameters then depends on the sample data. As we discussed, a more fully Bayesian approach would be to think of the scale for standardization as an unknown parameter to itself be estimated from the data.

P.S. Recall that "inputs" are not the same as "predictors."

P.P.S. I scale by 2 sd to be consistent with 0/1 predictors. In retrospect, I wish I'd scaled by 1 sd and then coded binary predictors as -1 and 1 to be consistent. This would've been simpler overall. But I think it's too late now.

(Right click on "View image" to see the whole thing.)

Among whites, vouchers appear to be more popular with the upper middle class and rich (with predictable religious variation: the strongest support is among Catholics, then born-again Protestants, then others).

Among blacks and hispanics, though, vouchers are more popular among the poor.

We'll have to check this on some other data.

Some details:

You can recognize celery in an x-ray scanner.

The other day while waiting for a bus, I was thinking about how city buses should be smaller and run more frequently. Instead of a 40-seater every 15 minutes, they could run a 10-seater every 5 minutes. (More precisely, they could run as frequently as necessary during rush hour to handle all the passengers--a bus a minute if necessary--but more spaced out at other times. For example, on weekend mornings the bus is never crowded, so they could run the much smaller buses with just slightly higher frequencies than they currently run big buses now.)

The advantages of my proposal are clear: the bus comes more frequently, also since the lag time is smaller, loading and unloading won't take so much time, and as an extra bonus, you'll probably skip a lot more stops because there are fewer people on the bus who might want to get off at any particular point. Also, I don't know about fuel efficiency, but I wouldn't be surprised if the fuel cost per passenger is lower because you're not having to run these huge empty buses in off-peak hours. Finally, van-sized buses could maneuver better in traffic.

The only additional cost that I see is having to hire more bus drivers, but with unemployment at 9%, I don't think it would be hard to find people to do this. What really irritates me are those huge, huge buses that take forever to fill up and take about a half hour just to go a few crosstown blocks. If they were broken up into vans, the wait would be less and the ride much more pleasant.

P.S. Yes, I know this isn't one of the world's most important problems. But it is a big expenditure, so why not try to do it right?

P.P.S. I'm sure there's lots of research on this topic but it's not something I'm at all informed on. The above are just my personal impressions.

P.P.P.S. To those of you who discuss the cost: Sure, it would cost money. But there's a real economic benefit: people would be able to get around the city faster! A good use of stimulus funds, etc.

I am at a conference which had an excellent poster session yesterday. I realized the session would have been even better if the students with posters had been randomly assigned to stand next to and explain other students' posters. Some of the benefits:

1. The process of reading a poster and learning about its material would be more fun if it was a collaborative effort with the presenter.

2. If you know that someone else will be presenting your poster, you'll be motivated to make the poster more clear.

3. When presenting somebody else's poster, you'll learn the material. As the saying goes, the best way to learn a subject is to teach it.

4. The random assignment will lead to more inderdisciplinary understanding and, ultimately, collaboration.

I think just about all poster sessions should be done this way.

P.S. In reply to comments:

- David writes that my idea "misses the potential benefit to the owner of the poster of geting critical responses to their work." The solution: instead of complete randoimization, randoimize the poster presenteres into pairs, then put pairs next to each other. Student A can explain poster B, student B can explain poster A, and spectators can give their suggestions to the poster preparers.

- Mike writes that "one strong motivation for presenters is the opportunity to stand in front of you (and other members of the evaluation committee) and explain *their* work to you. Personally." Sure, but I don't think it's bad if instead they're explaining somebody else's work. If I were a student, I think I'd enjoy explaining my tellow-students' work to an outsider. The ensuing conversation might even result in some useful new ideas.

- Lawrence suggests that "the logic of your post apply to conference papers, too." Maybe so.

At our sister blog, Lee discusses strategic retirement, or lack thereof, in the Supreme Court.

This is a good time for me to bring up my point that congressmembers and senators appear to decide make these decisions non-strategically, being more likely to retire when their party most needs them and their incumbency advantage and being less likely to retire when the could be replaced more costlessly. (For example, Frank Lautenberg running for reelection in 2008, a year when the Democrats could well have afforded a fresh face in a New Jersey senate race with little chance of losing.)

A couple days ago, I wrote, of Martin and Quinn's estimated positions of Supreme Court justices, that

I don't know whether to believe the numbers. Is the Anthony Kennedy of 2007 (ideology score 0.14) really so close to Hugo Black in 1970 (ideology score 0.06)? To look at it another way, according to these numbers, in 1973 (the year of Roe v. Wade), six of the justices are colored red and the median justice is listed at 0.67. In 2007, only five are red and the median is at 0.14. In fact, in 2005 the median is listed as -0.07, or slightly to the left of center. Is it really plausible that the court was more liberal in 2005 than in 1973? Maybe so, but something looks fishy to me here.

In reply, Andrew Martin wrote:

re: Black and Kennedy, I [Martin] tend to think of them as pretty similar. Both were moderates (although on somewhat different types of cases), one a moderate Dem the other a moderate Rep. So them being close is not implausible.

There were a couple of very liberal decisions in the 1972 term (when Roe was decided), including a Roe and a death penalty case. But even on civil liberties the court reached a conservative decision in Miller (the obscenity case). And there were some more conservative decisions in other areas of law. Today's court is surely more conservative on civil liberties issues (although there haven't really been many cases...), but may be a little to the left on some other issues (Hamdan). Today it all gets down to what Kennedy wants to do. If it were what Roberts would do the court would be far to the right.

That's an argument for plausibility, but the argument may be implausible. We tend to think about the Court in terms of the most politically salient cases, but the model treats Roe as equally to, say, a tax case. And, of course, the measures have huge limitations because they are just based on binary data, on all cases, with some reasonably strong model assumptions, etc.

The point about counting different domains of the law is interesting, along with the age-period-cohort sort of question of how you can even try to align left-right today with the corresponding positions in 1972.

Greg Mankiw looked up the Consumer Reports of ratings of car companies and found:

Dead last was Chrysler. CU recommended zero percent of the Chrysler vehicles they tested. That's right--zero. Second to last was General Motors. CU recommended 17 percent of GM models. By contrast, most other companies had half or more of their models get the thumbs up. Honda was the top ranked brand; CU recommended 95 percent of its models.

Mankiw writes:

Is it any surprise that Chrysler and GM are now in the process of going out of business? From the perspective of the Consumer Reports advice, it looks like their business model was to count on the ignorance of the buying public about the quality of their products. Their bankruptcy should perhaps be viewed as a success of the market system.

This makes sense to me, but I wonder if it explains too much. Presumably these companies have been making crappy cars for awhile. How did the companies stay alive so long? In all seriousness, perhaps the market system would've been more successful had it shut down those companies 10 or 15 years ago.

Beyond this is the principal-agent problem, or moral hazard, or whatever it's called, by which the people who make the decisions to make crappy cars are probably not actually going broke themselves: the companies might fall apart, but they'll do OK, I assume. So I can see how the companies could stay alive for awhile, living off their assets and their ability to borrow money. I just don't completely see it as a "success of the market" that they've been hanging on so long when the low quality of their products has been public knowledge.

The Numbers Guy has an article titled This U.K. Sheriff Cites Officials for Serious Statistical Violations, and a corresponding blog post:

Mobilized by distressingly low levels of public trust in official statistics, the U.K. government is embarking on a daring, and possibly unique, experiment. With broad support, Parliament in 2007 approved the formation of the U.K. Statistics Authority, a group with the budget, authority and independence to question other government agencies on the numbers they release to the public. [...]

The agency's task is a delicate one. If it uncovers reams of faulty data that might have been used in crafting public policy, Britons' fraying faith in public institutions could be further eroded.

Interesting, a truth-assurance agency would be a good thing, also useful for validating the truthfulness of other statements that often get twisted by marketing. We might be finally making progress with the problems that Josiah Stamp identified many years ago.

Pretty pictures (even if I don't believe the numbers being plotted); follow the link here.

1. Coalitions, voting power, and political instability.

Thurs 4 Jun, 3:30pm, Kane Hall 210 at the University of Washington. Part of the Math Across Campus series.

We shall consider two topics involving coalitions and voting. Each topic involves open questions both in mathematics (probability theory) and in political science.
(1) Individuals in a committee or election can increase their voting power by forming coalitions. This behavior yields a prisoner's dilemma, in which a subset of voters can increase their power, while reducing average voting power for the electorate as a whole. This is an unusual form of the prisoner's dilemma in that cooperation is the selfish act that hurts the larger group. The result should be an ever-changing pattern of coalitions, thus implying a potential theoretical explanation for political instability.
(2) In an electoral system with fixed coalition structure (such as the U.S. Electoral College, the United Nations, or the European Union), people in diferent states will have different voting power. We discuss some flawed models for voting power that have been used in the past, and consider the challenges of setting up more reasonable mathematical models involving stochastic processes on trees or networks.

2. Culture wars, voting and polarization: divisions and unities in modern American politics.

Fri 5 Jun, 9:45am, Kane Hall 225 at the University of Washington. Part of the 10th anniversary celebration of the Center for Statistics and the Social Sciences.

On the night of the 2000 presidential election, Americans sat riveted in front of their televisions as polling results divided the nation's map into red and blue states. Since then the color divide has become a symbol of a culture war that thrives on stereotypes--pickup-driving red-state Republicans who vote based on God, guns, and gays; and elitist, latte-sipping blue-state Democrats who are woefully out of touch with heartland values. But how does this fit into other ideas about America being divided between the haves and the have-nots? Is political polarization real, or is the real concern the perception of polarization? We address these questions using a results from our own research and that of others.

3. Creating structured and flexible models: some open problems.

Mon 8 Jun, 11am, Fairmont Lounge, St. John's College, 2111 Lower Mall, University of British Columbia. Statistics Department seminar.

A challenge in statistics is to construct models that are structured enough to be able to learn from data but not be so strong as to overwhelm the data. We introduce the concept of "weakly informative priors" which contain important information but less than may be available for the given problem at hand. We also discuss some related problems in developing general models for taxonomies and deep interactions. We consider how these ideas apply to problems in social science and public health. If you don't walk out of this talk a Bayesian, I'll eat my hat.

4. Red state, blue state, rich state, poor state.

Mon 8 Jun, 3pm, Fairmont Lounge, St. John's College, 2111 Lower Mall, University of British Columbia. Statistics Department seminar.

If you come to any of these, please ask lots of questions!

P.S. I've never spoken at UBC, but I have given a couple of talks in the statistics department at UW. The first time was twenty years ago. The talk went OK, I think--it was on medical imaging--but I did a horrible thing by leading off with a joke. I could probably get away with that now, but it didn't go over well then. In my defense, the joke was related to the topic of the talk. But it was a pretty bad joke. The second talk was about twelve years ago. The topic was model checking in spatial statistics. I think it went fine, but I recall that there was one spatial statistics expert in the audience who was disappointed at how simple my model was. It worked ok for what we were doing, though.

Drew Conway writes:

To paraphrase Bill James, the alternative to doing statistics is not "not doing statistics," it's "doing bad statistics."

Some people bemoan the excessive quantitative nature of academic political science nowadays. I certainly agree that there's room for nonquantitative work, but you also want to have some people who know their way around numbers. Or else you'll end up with this sort of horrible non-analysis by David Runciman of U.S. elections. What's striking about Runciman's article--and he's a well-respected political theorist, I'm sure--is that he relies on statistics all over the place. He just doesn't know what he's talking about--and, even worse, doesn't seem to know that he doesn't know.

I mouth off all the time about things I don't know about. But at least when I go on about Karl Popper, for example, I ground it in my own experience as a researcher, I don't just spout off in general.

Anyway, my point is not to pick on Runciman for a year-old article that he probably whipped off in a couple of hours and maybe already regrets. I'm just using it as an example of how people who don't know statistics are doomed to rely on statistics all the same.

Just as Bill James pointed out how fans who hate sabermetrics (and all it stands for) were forming all sorts of misinformed opinions based on batting averages and the like.

Chris Bowers writes:

The nation still moving away from Republicans demographically, too. It can't be emphasized enough that Michael Dukakis would have won the 2008 election. His exit polls of 40% among whites, 89% among African-Americans, and 70% among Latinos is enough to reach 50%+1 now, even in the event that African-American turnout was only 12% of the vote instead of 13%.

From our analysis of the Current Population Survey post-election supplement, here are our estimates for voter turnout in 2008: 76.4% white, 11.9% black, 7.4% hispanic, 4.3% other, with the categories defined as mutually exclusive (for example, if you're white and hispanic, you count as "hispanic"). The exit polls say 74% white, 13% black, 9% hispanic, and 5% other (not adding to 100% because of rounding error), but I think CPS is more trustworthy.

Now we can take the Dukakis numbers and plug them into the 2008 turnout numbers, as long as we make some estimate for the votes of "other." I'll assume 55%, halfway between his performance among whites and among hispanics. (By comparison, we estimate from the Pew pre-election polls that Obama got 45% of the two-party vote among whites, 96% among blacks, 68% among hispanics, and 59% among others.)

Plugging in Dukakis's percentages by ethnic group and using the turnout numbers of 2008, we get a national adjusted Dukakis vote of .40*76.4% + .89*11.9% + .70*7.4% + .55*4.3% = 48.7%, which is better than the 46.1% he actually received but not quite enough to win.

This doesn't really shoot down Bowers's main argument--demographic shifts are important. I think he was overstating his case just slightly.

And, yes, I know that if Dukakis had really been running in 2008, things would've been different. I'm just following Bowers in using the Dukakis vote as a handy way to summarize the trends, keeping voting by ethnic group constant. Voting by ethnic group is not constant (as we can see by comparing Obama's breakdowns to his predecessors), but doing this sort of calculation is a good way to visualize the demographic changes that are occurring.

Daniel Becker's Random-Walk graphically demonstrates how different distributions can be generated with physical processes: Normal distribution falls out of a Pachinko machine, and Poisson from a dart-throwing process. He also shows how pseudo random number generators have higher-order correlations within them. Pretty!

[via Infosthetics]

Robert Frank defends carbon offsets at the sister blog. I'm sympathetic to much of Frank's argument; in particular, the fact that Al Gore has a big house isn't much of an argument against carbon offsets. (If the crops are failing and the flood waters are rising, it won't be much help to stand on a street corner shouting: But Al Gore had a big house!)

But I'm not happy with the example that Frank chooses to illustrate his point. He writes:

Mojca has pointed me to Paul Nylander's visualizations. He's using raytracing software and Mathematica to create pieces of visualization art:

I tried looking for examples that could be useful in the usual statistical practice, and his example of Horseshoe magnetic fields demonstrates distributions on a surface:

Henry reports that a colleague of his at George Washington University, Jeffrey Rosen, says he has sworn off blogging for good. But then Henry asks whether Rosen was really a blogger at all: in Henry's words:

A 1,000 word commissioned essay for The New Republic, which goes through its usual editorial processes, is usually not considered a 'blog entry.' . . . I [Henry] will say that I'd prefer not to see the term blogpost become a residual category for 'stuff I wrote which I wish I had thought through a bit more before I hit send.'

I just have two comments on this intra-GW conflict:

1. Does blogging now have higher prestige than magazine writing? It used to be that newspaper and magazine writers were insisting that blogging wasn't journalism. Now we have unpaid bloggers saying that magazine writing isn't really blogging!

2. Given the New Republic's history, I wouldn't say that "its usual editorial processes" counts for much. I'm guessing I have a more rigorous vetting process on my own blog (where the rule is that anyone with access can post any time) than the New Republic has for its thousand-word commissioned essays.

Simon Jackman has some useful thoughts on the future of internet polling and why some of its critics don't know what they're talking about.

I'd also like to add that, as much as I rely on telephone sampling in my own research, I hate most of it in practice. Just yesterday, the phone rang and it was one of those robo-polls. I hung up, but I hated to even have to waste 5 seconds on the intrusion. I'm bothered by the asymmetry, that the survey wastes the time of the person who is being called while imposing a nearly zero cost on the pollster. Information pollution is what it is.

A few days ago I read Atul Gawande's article on health care costs and thought his story was interesting enough that I wanted to know the statistics on what factors predict high or low costs.

Commenter Marc pointed me to this recent article by Elliott Fisher, Julie Bynum, and Jonathan Skinner on regional variation in heath costs across the United States.

Commenter Ao pointed to this Congressional Budget Office report which, to me, was a bit disappointing. It had some nice maps and charts but did not seem nearly as serious as Gawande's article in trying to understand what was going on.

Finally, Alan Zaslavsky, a statistician who specializes in healh-care economics (and who uses multilevel models) wrote:

Atul Gawande's article in the New Yorker is an excellent review of some of the issues we have been struggling with in health policy research. While a lot of energy has gone into looking at the impact of various incentive schemes (public reporting of quality measures, "pay for performance") on quality of healthcare, it has been difficult to address the kinds of issues of organizational culture described in the article. The differences in provider structure and culture that are key to success or failure in providing high-quality, efficient care are not that readily brought into analysis -- the variables are just not measured and available. So instead we have very convincing case studies -- the McAllen market at one extreme and the integrated Kaiser, Geisinger etc systems at the other.

One problem is that many of the analyses take place at the level of the health (insurance) plan, but health plans in most cases are not in the business of providing care, they are in the business of buying it. (Even some of the original staff-model HMOs went through the transition to being insurance companies, like HIP in New York.) The variables that are routinely available for analysis at the health plan level are very crude proxies for the underlying organizational structures and cultures. For example, some economists have told me that they are perplexed by findings that not-for-profit plans provide better-quality care than for-profits, since both types of firms should be subject to similar incentives. What this leaves out is the distinct histories of some of the leading not-for-profits, and consequent differences in organizational cultures.

I do highly recommend the work of the Dartmouth group (including Jon Skinner and Elliott Fisher, mentioned in the article) on area variations to interested readers. However, it does suffer from some of the same limitations -- the variations can be found and clearly show that more is not always better, but it is hard to say what actually drives the differences or what can be done to implant the cultural and organizational features that would make more areas look like the best areas. [emphasis added]

While I [Zaslavsky] am a proponent of a universal system of health care, I don't think that will be enough to solve our problems without some fundamental changes in the incentives and structures under which care providers operate. Paradoxically, "rationalization" has meant squeezing out not the most wasteful aspects of care but some of the unprofitable but essential services that could make care more effective.

Perhaps more could be done to take the quantitative analyses of Fisher et al. and Zaslavsky and his colleagues, and see what is needed to move toward useful recommendations. This is all on top of the difficult political issues; for example, doctors in the U.S. get paid a lot and I don't think they'd be happy about getting less money.

In a review of a recent book by Edmund Morgan, Russell Baker mentions this story:

Two Boston carters [were] hauling a wagonload of wood along a narrow snow-drifted road one wintry day in 1705 when they met the oncoming coach of the royal governor of Massachusetts and refused to pull aside. The governor later testified that after he jumped from the coach and ordered them to clear the road, one of them "answered boldly, without any other words, 'I am as good flesh and blood as you; I will not give way, you may goe out of the way.'" When the governor drew his sword to assert respect for high office, the carter "layd hold on the governor and broke the sword in his hand."

It was "a supreme gesture of contempt for authority and its might," Morgan writes. One shudders to think what might happen to this stubborn working stiff and his supreme gesture in today's world of terrifying political motorcades with their heavily armed bodyguards.

Indeed.

The election of Barack Obama and the nomination of Sonia Sotomayor remind us of our society's great progress in social mobility, but in some ways we really do seem to be moving backwards.

David Friedman suggests that, instead of limiting a football team to 11 men, you allow the team as many men on the field as they'd like, with the constraint that their total weight be below some 2400 pounds. It's an interesting idea.

Commenters suggest the related idea of limiting the total height of a basketball team to 30 feet. Then we'd find out right away how tall these players really are.

I'm not saying these ideas are perfect, but they're interesting.

I was biking down the street and had to discard a banana peel. I approached a trash can and had to consciously decide to throw it at a point that seemed "too soon"--I was still several feet behind the receptacle. It went in.

What mystifies me is that the action was so unnatural. It really really felt like I should be throwing the banana peel just when I was going past the trash can. It was only my memories of physics class (and, I suppose, years of experience tossing things from a moving bike) that let me know when to release it--and, even then, it felt wrong.

I know that psychologists have done research on "folk physics"--the wrong intuitions we all have about heavy objects falling farther, ignorance of the action-and-reaction principle, and all the rest. But this thing with the banana peel is simple kinematics. You can't get much more basic than that. Yet my intuition remains out of whack. What's the deal?

Michael Kinsley goes over the top with this one:

It's clear that the one paralyzing fact about Sonia Sotomayor, to Republicans, is the color of her skin. If she weren't Latino, they would be in full revenge-for-Clarence-Thomas mode. Instead, they are in an agony of indecision, with GOP strategists openly warning: Support the Latina or die. If the 40 remaining Republican senators end up voting for Sotomayor, her race will be the reason.

Yes, congressional Republicans have been nearly unanimous in opposing Obama's economic plans. But that's no reason to be so sure they'd be unanimous in opposing a white Supreme Court justice. Just for example, Stephen Breyer was confirmed on an 87-9 vote, and if the Republicans wanted to be in "full revenge-for-Clarence-Thomas mode." that would've been a more natural time to do it. I know that Kinsley specializes in clever arguments, but in this case I think he's too clever by half.

To put it another way: the simplest response to Kinsley is that he's taking pure untestable speculation and claiming it's simply true. A more nuanced response is that I don't actually think his statement is true. Much depends on the particular nominee.

Amid an article about the GM bankruptcy, Mark Ambinder (political correspondent for the Atlantic Magazine) has the following offhand comment:

Purists -- and virtually every academic economist one happens to encounter -- wonder what happened to the once inviolate principle of rewarding risk-takers.

On a literal level, I don't think that's correct: the idea is that risk-takers can win big if they win, but if they lose, they lose: that's what "risk" is all about. I don't think anybody (except the risk-takers themselves, along with their friends and families) think that risk-takers should be rewarded when their bets lose.

But that's all obvious and fits in with all the moral-hazard, perverse-incentives things we've been hearing about for awhile.

Why I'm going on about this

What interests me is the centrality of "risk" in the world of economics now. Until being pointed to the article linked to above, I had never heard of the "once inviolate principle of rewarding risk-takers." Then again, it's been almost 30 years since I've taken an economics class. In that class, the idea of "risk" wasn't mentioned at all, I think. We learned about about supply and
demand, inflation and unemployment, money, investment, the stock market, etc. But the whole "risk" thing didn't come up.

Since then I've read enough to know that academic economists have been talking about
risk for awhile, but I don't think it was in the forefront of discussion. For example, I don't think a magazine columnist 30 years ago would've written about the inviolate principle of rewarding risk-takers, or anything of the sort. Things have changed--a lot.

I wrote that changes in the Hispanic vote wouldn't have had much effect on the 2008 election and, with the exception of Florida, might not be as important as people think in 2012 either.

Yair responds:

I don't know if I agree with your conclusions. It's true that no state truly "flipped," but many states became a lot more competitive and some become statistically indistinguishable from 50/50. Some specifics on this:

1. In the first scatterplot (excluding hispanics), you mention that NM, FL, IN, and NC come within 1% of switching and that changes to the model might make them flip. My interpretation is that this is a big change ... it would significantly alter the strategies of the campaigns, the media coverage, and everything else. It would also likely have an effect on other groups in the state, but this is tough to determine statistically and in fairness you assume everything else is held constant.

2. In the second scatterplot (halving hispanic McCain vote), TX AZ and GA all become competitive (as you mention), but also NM and to some extent FL become uncompetitive. Considering the number of electoral votes in TX and FL, this is potentially a massive change ... imagine if Democrats no longer had to worry about FL and Republicans started having to put a ton of money to keep TX's 34 electoral votes. This becomes even more pronounced in the final scatter (increasing 20% and halving McCain).

The point is, even though no states truly flipped, the playing field would hugely shift and have a ton of consequences on the campaign. In the post you use the assumption that nothing else would change, but my guess is that a less academic audience (i.e. 538) wouldn't like this assumption (I haven't read the comments yet). Because of this and the fact that there is modeling error, my interpretation is that these changes are more meaningful than described.

Good points; see also my other thoughts here.

Here.

Nate asks, "Can the Republicans win back the White House in 2012 or 2016 while losing further ground among Latinos?" I don't know about 2012 and 2016, but I can give my best estimates for 2008, based on my analysis with Yair using the Pew pre-election polls to get vote preferences (normalizing each state to line up with the actual election outcome) and the CPS post-election supplement to get voter turnout. (You'll get similar numbers using the exit polls, but I trust our analyses a little more, also they're consistent with our earlier graphs of voting by income and ethnicity.)

I'll show you what we found, then give some brief discussion.

Here's how Obama did among Hispanics in the states where there is a large Hispanic presence:

[In response to commenters, here are some numbers for McCain's estimated share of the two-party vote among Hispanics: NM 27%, CA 26%, TX 42%, FL 43%, AZ 35%, NV 24%, NY 25%, CO 27%, NJ 23%, IL 23%, CT 24%. Exit polls give slightly different answers. No data source is perfect and we have to acknowledge that there is uncertainty in our estimates.]

And here's a map showing our estimate of the Hispanic vote share by state (based on the CPS post-election supplement): Hispanics represented 31% of the vote in New Mexico, 22% in California, 20% in Texas, 15% in Florida, 13% in Arizona, 12% in Nevada, and less than 10% in all other states:

OK, so Obama dominated among Hispanics. How did he and McCain do among the rest of the voters? The following map shows our estimates from our model based on the Pew data:

This map looks suspiciously close to the map for all voters. And, in fact, it is.

Could someone build me a physical version of this?

Atul Gawande wrote an interesting article about health-care costs, focusing on McAllen, Texas, which he describes as "one of the most expensive health-care markets in the country. . . . In 2006, Medicare spent fifteen thousand dollars per enrollee here, almost twice the national average." In some ways, Gawande's article is like a case-control regression analysis without the numbers: he compares McAllen to the national average and to various other places in the United States, and looks the similarities and differences to find systematic patterns. He concludes that the key problem is "untenably fragmented, quantity-driven systems of health care," in which doctors are motivated to do more and more, with no apparent beneficial effects on the patients.

What do the experts say?

I imagine this is an area where health-economics statisticians have done some research. I'd be interested to hear the comments of Sharon-Lise Normand, Alan Zaslavsky, or some other expert in this field. They very well may have run some regression analyses to try to understand the factors that explain variation in health care costs at the regional, state, and local levels.

Smoking

As a minor point, I was puzzled by an offhand comment that Gawande made:

An unhealthy population couldn't possibly be the reason that McAllen's health-care costs are so high. (Or the reason that America's are. We may be more obese than any other industrialized nation, but we have among the lowest rates of smoking and alcoholism, and we are in the middle of the range for cardiovascular disease and diabetes.)

I don't know how things go with alcoholism, but my impression of smoking was that it caused a net decrease in health care costs: smokers tend to die younger, and to die quickly once they get seriously ill, thus sparing the health care system some of the big-ticket end-of-life costs. For example, from this 1997 article by Barendregt et al. on the health care costs of smoking:

Health care costs for smokers at a given age are as much as 40 percent higher than those for nonsmokers, but in a population in which no one smoked the costs would be 7 percent higher among men and 4 percent higher among women than the costs in the current mixed population of smokers and nonsmokers. . . . If people stopped smoking, there would be a savings in health care costs, but only in the short term. Eventually, smoking cessation would lead to increased health care costs.

Yuck.

Time moves fast in the blogosphere. This particular discussion started four days ago when Robin wrote:

Aleks points me to this cool-looking automatic dice thrower. I like it!

Stephen Senn quips: "A theoretical statistician knows all about measure theory but has never seen a measurement whereas the actual use of measure theory by the applied statistician is a set of measure zero."

Which reminds me of Lucien Le Cam's reply when I asked him once whether he could think of any examples where the distinction between the strong law of large numbers (convergence with probability 1) and the weak law (convergence in probability) made any difference. Le Cam replied, No, he did not know of any examples. Le Cam was the theoretical statistician's theoretical statistician, so there's your answer.

The other comment of Le Cam's that I remember was his comment when I showed him my draft of Bayesian Data Analysis. I told him I thought that chapter 5 (on hierarchical models) might especially interest him. A few days later I asked him if he'd taken a look, and he said, yes, this stuff wasn't new, he'd done hierarchical models back when he'd been an applied Bayesian back in the 1940s.

A related incident occurred when I gave a talk at Berkeley in the early 90s in which I described our hierarchical modeling of votes. One of my senior colleagues--a very nice guy--remarked that what I was doing was not particularly new; he and his colleagues had done similar things for one of the TV networks at the time of the 1960 election.

At the time, these comments irritated me. But, from the perspective of time, I now think that they were probably right. Our work in chapter 5 of Bayesian Data Analysis is--to put it in its best light--a formalization or normalization of methods that people had done in various particular examples and mathematical frameworks. (Here I'm using "normalization" not in the mathematical sense of multiplying a function by a constant so that it sums to 1, but in the sociological sense of making something more normal.) Or, to put it another way, we "chunked" hierarchical models, so that future researchers (including ourselves) could apply them at will, allowing us to focus on the applied aspects of our problems rather than on the mathematics.

To put it another way: why did Le Cam's hierarchical Bayesian work in the 1940s and my other colleague's work in 1960s not lead to more widespread use of these methods? Because these methods were not yet normalized--there was not a clear separation between the math, the philosophy, and the applications.

To focus on a more specific example, consider the method of multilevel regression and poststratification ("Mister P"), which Tom Little and I wrote about in 1997, then David Park, Joe Bafumi and I picked back up in 2004, and then finally took off with the series of articles by Jeff Lax and Justin Phillips (see here and here). This is a lag of over 10 years, but really it's more than that: when Tom and I sent our article to the journal Survey Methodology back in 2006, the reviews said basically that our article was a good exposition of a well-known method. Well-known, but it took many many steps before it became normalized.

Here.

P.S. I added a note explaining why Mankiw's reasoning may make more sense than I thought at first.

P.P.S. Update here.

Edo Airoldi writes:

We have two postdoctoral fellowships available for up-to three years, with competitive salary and travel support. The two postdocs are expected to contribute to active research projects, including:

1. Development of statistical methodology, algorithms, and theory for analyzing complex graphs and dynamical systems,

2. Analysis of coordinated regulatory mechanisms driving the cell cycle, metabolism, and environmental responses in yeast, bacteria, and cancer systems,

3. Analysis of signaling and metabolic pathways in dynamic chemical contexts, and

4. Development of dynamic models of coalition formation and stability.

Somehow this all reminds me of a hilarious Veronica Geng piece from 1986 where she riffs on a comment of Barry Goldwater's that "There's no reason we shouldn't have an Italian President--we've had everything else." She goes through a bunch of examples, including "President Thomas Noguchi (the only coroner ever to become Chief Executive)" and the time when "an obscure Pennsylvania coal miner named James Polki got elected President until the Electoral College found out a novice telegraph operator had made a mistake--if you want to call it a 'mistake' that American history includes a hardworking Polish immigrant who held the highest office in the land, temporarily," and, of course, "the way a woman named Dora, or Doreen, occupied the Oval Office in 1920 as a poltergeist." As Geng puts it, "It's just an open secret--like the fact that George Washington was a full-blooded Chickahominy Indian, which everybody knows, even though it never appears in print anywhere."

To continue a discussion from a couple days ago . . . Robin Hanson, an economist who has written about his goal of overcoming bias and obtaining beliefs closer to reality, wrote something recently about "the signaling persona behind common ideologies," in this case, libertarian, conservative, and liberal political attitudes. What struck me when reading Robin's article was how far off from reality his descriptions of conservatives and liberals seemed to be.

I have some further thoughts here.

Michael Maltz wrote an excellent article on visualizing data in criminal justice. You can read the article yourself, but I just wanted to comment on something he writes on page 12 of his article: "Promote Data Visualization as a First Step." I completely agree--and he has some good examples to make this point--but I'd also like to promote data visualization as an intermediate step (to check fit of model to data) and a last step (to summarize the inferences from a fitted model).

These are all important methods and concepts related to statistics that are not as well known as they should be. I hope that by giving them names, we will make the ideas more accessible to people:

Mister P: Multilevel regression and poststratification.

The Secret Weapon: Fitting a statistical model repeatedly on several different datasets and then displaying all these estimates together.

The Superplot: Line plot of estimates in an interaction, with circles showing group sizes and a line showing the regression of the aggregate averages.

The Folk Theorem: When you have computational problems, often there's a problem with your model.

The Pinch-Hitter Syndrome: People whose job it is to do just one thing are not always so good at that one thing.

Weakly Informative Priors: What you should be doing when you think you want to use noninformative priors.

P-values and U-values: They're different.

Conservatism: In statistics, the desire to use methods that have been used before.

WWJD: What I think of when I'm stuck on an applied statistics problem.

Theoretical and Applied Statisticians, how to tell them apart: A theoretical statistician calls the data x, an applied statistician says y.

The Fallacy of the One-Sided Bet: Pascal's wager, lottery tickets, and the rest.

Alabama First: Howard Wainer's term for the common error of plotting in alphabetical order rather than based on some more informative variable.

The USA Today Fallacy: Counting all states (or countries) equally, forgetting that many more people live in larger jurisdictions, and so you're ignoring millions and millions of Californians if you give their state the same space you give Montana and Delaware.

Second-Order Availability Bias: Generalizing from correlations you see in your personal experience to correlations in the population.

The "All Else Equal" Fallacy: Assuming that everything else is held constant, even when it's not gonna be.

The Self-Cleaning Oven: A good package should contain the means of its own testing.

The Taxonomy of Confusion: What to do when you're stuck.

The Blessing of Dimensionality: It's good to have more data, even if you label this additional information as "dimensions" rather than "data points."

Scaffolding: Understanding your model by comparing it to related models.

Ockhamite Tendencies: The irritating habit of trying to get other people to use oversimplified models.

Bayesian: A statistician who uses Bayesian inference for all problems even when it is inappropriate. I am a Bayesian statistician myself.

Multiple Comparisons: Generally not an issue if you're doing things right but can be a big problem if you sloppily model hierarchical structures non-hierarchically.

Taking a model too seriously: Really just another way of not taking it seriously at all.

God is in every leaf of every tree: No problem is too small or too trivial if we really do something about it.

As they say in the stagecoach business: Remove the padding from the seats and you get a bumpy ride.

Story Time: When the numbers are put to bed, the stories come out.

The Foxhole Fallacy: There are no X's in foxholes (where X = people who disagree with me on some issue of faith).

The Pinocchio Principle: A model that is created solely for computational reasons can take on a life of its own.

The statistical significance filter: If an estimate is statistically significant, it's probably an overestimate.

Arrow's other theorem (weak form): Any result can be published no more than five times.

Arrow's other theorem (strong form): Any result will be published five times.

I know there are a bunch I'm forgetting; can youall refresh my memory, please? Thanks.

P.S. No, I don't think I can ever match Stephen Senn in the definitions game.

Robin Hanson posted a discussion of the differences between liberals, conservatives, and libertarians in which he considers not just their disagreements on issues, but their differences in who they respect.

I have to admit I've never really had a clear understanding of what "libertarian" means; perhaps because they have never been in power, it's more difficult to pin down a particular set of attitudes and positions for them.

But what I wanted to talk about here are Hanson's descriptions of conservatives and liberals, which seem to me to illustrate the difficulties of trying to understand, even sympathetically, views much different from one's own.

A couple of weeks ago, I received an email that began, "Only in crazy academia land" . . .

And then today I was reading an article by David Denby in the New Yorker about the director of The Wizard of Oz, and came across this:

Academics have told me [Denby] that "Oz" is a mythic structure, a descendant of the Odyssey or the Aeneid, but they look at me blankly when I say that the movie is also a summa of nineteen-thirties show business.

What kinds of academics was he talking with?? I'm no expert, but even I know that Dorothy's sidekicks were old vaudevillians.

Denby goes on about how great The Wizard of Oz is. I think it's ok, but I bet that much of its fame arises from it being shown once every year on TV. When we were kids, we would watch it every year. I'm guessing that there are a bunch of other movies made around the same time that are just great, but they didn't get that kind of exposure.

John Sides and I wrote an op-ed for the Daily News:

Daniel Carlat posts a link to this news article by John Fauber about a medical researcher, James Stein, who took big bucks in lecturing and consulting fees from drug companies over a 12-year period, before stopping a few months ago. Stein said:

I was sure I could avoid bias because I controlled the content and I had these strong personal convictions. Well, unfortunately, over the past several months, I've learned that I was wrong. I've learned that I could not stay unbiased, that I could not control all the content of my talks, and that my personal convictions were not good enough.

Regarding disclosure as a potential solution, Stein said:

I really felt that if I stood up in front of a crowd and said that these are my disclosures, look how honest I am, that I was really managing conflict of interest. But actually the medical literature and the social science literature tells me that it is actually the opposite effect. Although it is laudable to disclose your relationships, actually thinking that disclosure manages relationships is harmful. It has the perverse effect that when you disclose your relationship, the recipient of your information becomes more trusting, and the social scientists also have shown us that professionals who disclose actually become more biased.... I would argue ... that the solution is not disclosure, because if you are doing something that is wrong or unethical, don't disclose, just don't do it!

There was also this amazing bit:

Huge fines or convictions for gross ethical conduct were being issued against every drug company that he worked with. Doctors were being investigated on allegations of taking kickbacks.

Hal Daume pointed me to this. I haven't tried it out yet, but it looks like the right idea.

Catherine Rampell posted some attractive county-level Human Development Index maps and also discussed my criticisms of the index: I wrote, "if you go by the maps that everybody's linking to...you're pretty much just mapping state income and giving it a fancy transformation and a fancy new name." In its defense, she wrote:

Which is, I [Rampell] suppose, why the American Human Development Index, an adapted version of the U.N.'s original H.D.I., was created: because the U.N.'s index was not designed to capture the levels of variation that would occur within a single country. It was designed to make international comparisons.

This, to me, indicates the problem with the index. It was advertised as putting U.S. states on an international scale (Louisiana vs. Croatia and all that) but, if it needs to be redefined for the U.S., it seems to me that you're losing the universal interpretation, which is a big justification for the index in the first place. At this point, I'd rather map each of the components of the index separately (as Rampell actually does illustrate on her blog).

From comments to my recent 538 post, I've learned the following:

1. I don't understand logarithms.

2. I really don't understand logarithms.

3. I don't know how to use Wikiipedia.

4. I don't know that "percent" means "divided by 100".

5. I don't know the difference between correlation and R-squared.

My first reaction is to respond in a snippy and sarcastic way, but when it comes to writing, the reader is (almost) always right: When someone misunderstands something I wrote, this tells me I was being unclear.

Greg Mankiw reports on an article by Betsey Stevenson and Justin Wolfers that finds:

By many objective measures the lives of women in the United States have improved over the past 35 years, yet we show that measures of subjective well-being indicate that women's happiness has declined both absolutely and relative to men. . . . Relative declines in female happiness have eroded a gender gap in happiness in which women in the 1970s typically reported higher subjective well-being than did men. . . .

Mankiw concludes: "It sounds like either the women's movement was a mistake or subjective happiness is not the right objective." The bit about the women's movement doesn't make sense to me--this reasoning seems to contradict the point Mankiw made a few days ago about the difficulty of making inferences based on n=1.

If I had to make a quick guess, I would've gone with the hypothesis of economic stress combined with the difficulty of having a job and taking care of the kids, but Stevenson and Wolfers discuss this issue (see pages numbered 15 and 17 and Table 3 of the linked article) and show that the data don't particularly support this hypothesis.

Getting back to Mankiw's comment: Setting aside the line about the women's movement--who knows, maybe the women's movement was a mistake, it's hard to say with n=1 what might have happened in its absence--I think he's right that subjective happiness is not an "objective." People have written about this: you don't become happy by aiming for happiness as an objective, you become happy by doing things that make you happy (or, just by being the kind of person who's happy in any case). It's an interesting issue, but I'm not sure how this is relevant to the Stevenson and Wolfers study.

P.S. If I were Betsey Stevenson, I might be a little unhappy that Mankiw referred to the authors unalphabetically as Wolfers and Stevenson!

P.P.S. Mankiw has fixed this and put the authors in the correct order.

See the end of this entry for explanation.

Alex Hoffman pointed me to this widely-circulated map comparing the fifty states in something called the Human Development Index:

As Alex points out, the coding of the map is kind of goofy: the states with the three lowest values are Louisiana at .801, West Virginia at .800, and Mississippi at .799, but their color scheme makes Mississippi stand out as the only yellow state in a sea of green.

But I'm concerned about more than that. Is Alaska really so developed as all that? And whassup with D.C., which, according to the table, is #4, behind only Connecticut, Massachusetts, and New Jersey? I know about gentrification and all that, but can D.C. really be #4 on any Human Development Index worth its name?

Time to look behind the numbers.

I just got this unsolicited email:

Commenters pointed out that the map to which I linked yesterday actually shows the number of people entering each station, not, as implied by the visual structure of the map, the traffic on the subway lines between the stations. I agree with the commenters that line width doesn't seem like a good way to show information that is at the station level. Better to use differently-sized circles or something like that.

But this sets up a fun statistical problem: estimate the traffic on the subway lines given the data on the number of people entering each station (along with any other available data, and whatever modeling assumptions are needed to complete the picture). I guess there must be people at the transportation dept. doing this sort of thing, but I wouldn't be surprised if they're using deterministic solve-for-x algorithms that could be improved by a more statistical approach.

P.S. Richard Clegg writes in:

As you surmised this is a well-studied problem. Actually in the field of road transport this would be broken into two separate but related problems -- the origin-demand matrix estimation problem (given a set of observations what set of demands from origin to destination best explain them) and the related traffic assignment problem (given an origin demand matrix and a network with limited capacity on links how does one assign traffic onto network links).

In particular the traffic assignment problem has some attractive statistical properties if certain assumptions are made.

I replied:

About 25 yrs ago I worked on finite-element methods for thermal models, so I figured the mathematics would be similar. As noted on blog, I suspect that inclusion of some stochastic elements to the problem could improve things as well as extend the range of problems to which these methods could be applied.

And Clegg added the following:

For the origin-demand matrix problem there are a variety of approaches both frequentist and Bayesian -- I am far from an expert here (but hope to be more expert soon since I am involved with a grant proposal on the subject which I am hoping will be funded). For the traffic assignment problem there are a number of approaches, "deterministic" and "stochastic" to varying degrees. In the stochastic approach you make certain assumptions about how users disperse across routes of different costs (by assuming an error distribution on the user's perception of route costs -- as it turns out, a Gumbell distribution often produces "nice" answers). There are even the so-called "doubly stochastic" problems where the demand from each origin to each destination is assumed to have a distribution and then users perceives routes imperfectly according to another distribution. If you google "Stochastic user equilibrium" you will find more about the problem than you ever wanted to know.

Sounds good. I also expect there's some room for improvement using hierarchical modeling.

Aleks points me to this map (link from here) showing subway ridership using line widths.

It's fun, and it would be good to do something similar with road traffic using available data. The statistical problems would be interesting too: road traffic data are incomplete, so you'd want to do some modeling to get reasonable numbers over time. This could be a great project, actually.

My only comment on the subway graphs is that the B and D trains seem to have disappeared below 145th St.:

Maybe their lines could be run parallel to the A/C on the graph? There's a similar problem with the E/F in Queens. Also it looks to me like two of the 1/2/3 lines have disappeared below 96th St.

P.S. More here.

Eric Gilbert and Karrie Karahalios have a paper on tie strength, distinguishing between strong and weak ties in social networks, published at the Computer and Human Interaction conference. Eric is one of the recipients of 2009 Google fellowships. There are some neat ideas there:

Presenting the distributions of predictors

Pretty, informative and compact.

Distribution of outcomes

Not sure the median is particularly interesting.

Graphical model summary

They describe it as:

The predictive power of the seven tie strength dimensions. [...] A dimension's weight is computed by summing the absolute values of the coefficients belonging to it. The diagram also lists the top three predictive variables for each dimension. [...]

While the aggregation of coefficients in the same category is nice, there are some problems summing betas together. Rarely occurring values with huge betas are often an artifact of overfitting and not of informativity, and betas for continuous predictors are strongly affected by scale. Consider these betas:

So, the top two predictors are probably correlated, and opposite to one another - resulting in runaway absolute betas.

I've suggested the concept of net leverage a few years ago in a natural language binary outcome setting as an attempt to improve the presentation of feature importance in regression models, but this topic is worth revisiting.

I wonder what Seth thinks about this one. The blurb says, "It gets your natural biology working for you - rather than against you, as it so often does - and has been endorsed by an unprecedented number of top weight control experts." Also, it talks about your natural instincts and it's designed by someone named Roberts who is a Ph.D. researcher at a major university. Lots of similarities to Seth's diet so far.

The one irritating thing about it is that I can't figure out exactly what the diet plan is. The information on the website is vague--I can understand this, if they give out the information for free, maybe they won't sell as many books--but it's a bit more annoying to see this vagueness in journalistic descriptions. For example, this article in the "Daily Beast" (?) basically advertises the diet without giving any details. What's that about? Can't the reporter buy the book, read it, and tell us what it's about? Otherwise, what's the point.

I'm not actually familiar with the Daily Beast; from its main page, it appears to be a news/celebrity site, sort of like Radar without the humor? I can't tell if their article on the diet plan is an intentional plug--part of some advertorial system--or maybe it's just that the reporter thought that readers would be more interested in a puff than in a description of how the diet actually works (beyond that it involves lots of recipes).

Greg Mankiw has a nice little discussion of the difficulty of evaluating the effects of interventions in the n=1 setting:

As Mankiw points out, the bad news about the unemployment rate is bad news with or without the recovery plan and thus--although it certainly seems to knock down the predictions shown in that graph--it does not provide much information on the causal effect of the fiscal stimulus. Especially given that the graph comes from a report released in early January, before anyone knew what would end up being included in the final version of the stimulus plan.

This seems kind of horrible:

Fresh out of ideas on what to write about? Your blog muse taking a holiday? Don't sweat it. Ideas are everywhere; you just have to know where to look. We've rounded up ten suggestions to help you uncover a wealth of new topics and blog another day. . . .

That's all we need . . . people blogging who have nothing useful to say!

From the Polmeth mailing list:

I've made this point before, but I just received an email on the topic and so I thought I'd point youall to section 3.3 of this article of mine from 2003 where I make the argument in detail.

This article--A Bayesian Formulation of Exploratory Data Analysis and Goodness-of-fit Testing--is one of my favorites. It also features:
- A potted history of Bayesian inference (section 2.1)
- The first published definition (I think) of U-values and P-values (section 2.3)
- A model-checking perspective on the problem of degenerate estimates for mixture models (section 3.1)
- Why this isn't all obvious (section 5)

The article is based on a presentation I gave a year earlier at a conference. It was supposed to appear in the proceedings volume, but it was late, and the conference organizer was so annoyed he refused to include it. So I published it in the International Statistical Review instead. A year later I published a related article, Exploratory Data Analysis for Complex Models, as a discussion paper in the Journal of Computational and Graphical Statistics. That second article is more coherent, but personally I prefer the International Statistical Review article because it covers so many little topics that don't fit into existing theories of inference. I think of these examples as analogous to the quantum anomalies that toppled classical physics around 1900. In this case, what I want to topple is classical Bayesian inference--by which I mean Bayesian theory that does not include model building and model checking.

Groves rules out use of sampling in 2010 census:

President Barack Obama's pick to lead the Census Bureau on Friday ruled out the use of statistical sampling in the 2010 head count, seeking to allay GOP concerns that he might be swayed to put politics over science. Robert M. Groves, a veteran survey researcher from the University of Michigan, also testified during his confirmation hearing that he remains worried about fixing a persistent undercount of hard-to-reach populations . . . Census officials have already acknowledged that tens of millions of residents in dense urban areas -- about 14 percent of the U.S. population -- are at high risk of being missed because of language problems and an economic crisis that has displaced homeowners.

My comments:

I have a great respect for Bob Groves, and I would trust his decisions on what to do with the Census more than I would trust my own.

Bob's statement that "there is simply no time to prepare for it" seems eminently reasonable to me, especially given the cost constraints under which the census operates. On the statistical merits of the issue, I'm pretty sure that adjusted numbers would be better than unadjusted numbers. The census people know what they're doing, and there are known problems of nonresponse, and, for anything where I care about the damn answer, I'd use their adjusted estimates over the raw numbers.

As a social scientist, I hope the census bureau could release two sets of numbers, one unadjusted for political reasons and one adjusted for those of us who want the most accurate inferences possible.

That said, I'm ignoring a possible indirect effect of adjusting the numbers: If people know that the census will do adjustment, maybe they'll be less likely to participate in the enumeration in the first place. It's hard to measure such an effect and, hey, it might be important. I don't know.

I'm not thinking so much of individuals deciding whether to respond to the census, but rather of the decisions of local jurisdictions, where various spending formulas depend on population. For example, if it's known that the census won't be adjusted, then I'd expect the government of New York City to put a lot of effort into convincing people to participate. If it is known that the census will be adjusted, then there'd be a lot less motivation for localities to do what it takes to boost participation.

Conditional on the data already being collected, you'd definitely want to make statistical adjustments; it's a tougher call to decide on this ahead of time. Also, if you know for sure you won't be adjusting, this will affect the effort you put into collecting the data in different places. So if you're not going to adjust, you might as well make that decision right away.

P.S. To expand on this slightly, I think any debates over census adjustments are fundamentally political debates, not statistical disagreements. The scientific consensus on adjustment is pretty easy (although people can argue about the details of implementation, as noted by Lawrence in comments below). It's the political consensus that's difficult, as there are clear winners and losers. With a lack of political consensus, all you need is a little bit of dust and confusion in the air to give a sense of a lack of scientific consensus, which then gets piped back in to justify inaction in the political process.

Yesterday I showed Laura Wattenberg's graphs showing the most popular last letters of boys' names in 1906, 1956, and 2006. The quick story is that 100 years ago, there were about 10 last letters that dominated; 50 years ago, the number of popular last letters declined slightly, to about 6; but now, a single letter stands out: an amazing 36% of baby boys in America have names ending in N.

This is super-cool. As a commenter wrote, there should be some sort of award for finding the largest effect "in plain sight" that nobody has noticed before.

But, beyond pure data-coolness, what does this mean? My story, based on reading Wattenberg's blog, goes as follows:

100 years ago, parents felt very constrained in their choice of names (especially for boys). A small set of very common names (John, William, etc.) dominated. And, beyond that, people would often choose names of male relatives. Little flexibility, a few names being extremely common, resulting in a random (in some sense) distribution of last letters.

Nowadays, parents have a lot of freedom in choosing their names. As a result, there are lots and lots of names that seem acceptable, but the most common names are not so common as they were fifty or a hundred years ago. With so much choice, what do people do? Wattenberg suggests they go with popular soundalikes (for example, Aidan/Jaden/Hayden), which leads to clustering in the last letter. Even so, the pattern with N is so striking, there's gotta be more to say about it.

But I like the paradox: 100 years ago, the distribution of names was more concentrated but the distribution of sounds (as indicated by last letters) was broader. Nowadays, the distribution of names is more diffuse but the distribution of sounds is more concentrated.

Less constraint -> more diffuse distribution of names -> more concentrated distribution of last letters.

This must occur in other aspects of life. For example, consider food. We eat lots more different types of food than we used to, but a single ingredient--corn syrup--makes up more and more of our diet (or so I'm told). Again, lack of constraint (this time for economic reasons) leads to more diversity in some ways and more homogeneity (by choice) in others.

Andrew J. Oswald and Nattavudh Powdthavee write:

In remarkable research, the sociologist Rebecca Warner and the economist Ebonya Washington have shown that the gender of a person's children seems to influence the attitudes and actions of the parent.

Warner (1991) and Warner and Steel (1999) study American and Canadian mothers and fathers. The authors' key finding is that support for policies designed to address gender equity is greater among parents with daughters. This result emerges particularly strongly for fathers. Because parents invest a significant amount of themselves in their children, the authors argue, the anticipated and actual struggles that offspring face, and the public policies that tackle those, matter to those parents. . . The authors demonstrate that people who parent only daughters are more likely to hold feminist views (for example, to favor affirmative action).

By collecting data on the voting records of US congressmen, Washington (2004) is able to go beyond this. She provides persuasive evidence that congressmen with female children tend to vote liberally on reproductive rights issues such as teen access to contraceptives. In a revision, Washington (2008) argues for a wider result, namely, that the congressmen vote more liberally on a range of issues such as working families flexibility and tax-free education.

Our [Oswald and Powdthavee's] aim in this paper is to argue, with nationally representative random samples of men and women, that these results generalize to voting for entire political parties. We document evidence that having daughters leads people to be more sympathetic to left-wing parties. Giving birth to sons, by contrast, seems to make people more likely to vote for a right-wing party. Our data, which are primarily from Great Britain, are longitudinal. We also report corroborative results for a German panel. Access to longitudinal information gives us the opportunity -- one denied to previous researchers -- to observe people both before and after they have a new child of any particular gender. We can thereby test for political 'switching'. Although
panel data cannot resolve every difficulty of establishing cause-and-effect relationships, they allow sharper testing than can simple cross-section data.

They addressed the concerns about the research I'd expressed earlier.

Just one thing . . .

I have only one request, and I know it's too late because the article is already scheduled to appear in a journal, but I'll ask anyway. The article has lots of graphs and lots of tables--and I'll spare you my detailed thoughts on these, because, again, it's already scheduled to appear.

But one thing that I didn't see graphed is what I would think is the most natural and important thing to graph: the estimated change in the probability of voting for the conservative party, comparing a parent of a boy compared to the parent of a girl. That is, the estimated effect on the vote of having a boy, compared to a girl. I assume this effect varies by sex and age of parent and also by age, number of previous children, past voting patterns, and other factors.

Earlier today we had a brief discussion here about interactions, and this is a great example for thinking about modeling and displaying them.

(The graphs that are in the Oswald and Powdthavee article give average numbers of boys and girls for voters of different parties, but that's not quite what I'm looking for. As the authors so clearly explained, the key question is the effect of the sex of the child on parents' attitudes and behavior, and I'd like a graph that would really show this. As it is, I honestly have difficulty figuring out the estimated effect size here. Yes, it's great to see that coefficients are statistically significant--but I want to see what's going on here. I want to see the estimated effect.)

Is this for real?

1906:

1956:

2006:

Wow.

Andrew Grogan-Kaylor writes:

With John Yoo writing for the Philadelphia Inquirer, it seems worth recycling these thoughts from last year, on the unfortunate occasion of Yoo writing a botched column about elections of the early 1800s. As I wrote last year:

Buh-bye.

Interesting comments too.

My former Columbia colleague Matt Kahn sent me this article by Michael Cragg and himself on the political economy of congressional support for legislation intended to mitigate greenhouse gas production:

Stringent regulation for mitigating greenhouse gas emissions will impose different costs across geographical regions. Low-carbon, environmentalist states, such as California, would bear less of the incidence of such regulation than high-carbon Midwestern states. Such anticipated costs are likely to influence Congressional voting patterns. This paper uses several geographical data sets to document that conservative, poor areas have higher per-capita carbon emissions than liberal, richer areas. Representatives from such areas are shown to have much lower probabilities of voting in favor of anti-carbon legislation. In the 111th Congress, the Energy and Commerce Committee consists of members who represent high carbon districts. These geographical facts suggest that the Obama Administration and the Waxman Committee will face distributional challenges in building a majority voting coalition in favor of internalizing the carbon externality.

They make some interesting points, somewhat related to the much-remarked issue that the Democratic-leaning northern and midwestern states tend to pay more in taxes than they get back in government spending, while Republican-leaning sunbelt states are generally net beneficiaries of federal funds. When looked at from this perspective, you can see it's not so simple as Democrats vs. Republicans. Also, is straight carbon emissions the only story? I see from the map that Michigan has low carbon emissions per capita, but, at least traditionally, the politicians there support heavy industry. I suppose that, nowadays, carbon emissions is much more about extraction than about industrial production.

Cragg and Kahn do an analysis at the congressional district level, which makes a lot of sense. I haven't looked at income and voting by congressional district, but when you look at it by county, the patterns vary a lot by state. In California, Washington, and Oregon, the richer counties are nowadays the most Democratic. But in Texas and Oklahoma, the pattern goes the other way, with richer counties being more Republican. For example, suburbs of Dallas. So I think you have to be careful about using phrases such as "conservative, poor areas" and "liberal, richer areas." This pattern fits some parts of the country but not others (a point we made ad nauseum in Red State, Blue State). I think I know what Cragg and Kahn mean by this--they mean that, when they run a regression, both the average liberalness and the average income in the congressional district predicts lower carbon emissions--but you're just asking for trouble if you blur these concepts.

The other thing I wonder is if Cragg and Kahn have fully accounted for the partisan nature of Congressional voting. To put it bluntly: the Democrats have a majority in both houses of Congress, and so their votes count more than the Republicans'. This should affect their analysis and conclusions. On pages 17-18, they do discuss differences between the parties, but unless I'm missing something (and maybe I am), they're downplaying the relevant fact that the Democrats are in the driver's seat.

I also have a few comments about the data display (of course):

David Weakliem and I wrote an article that will be appearing in American Scientist, which is a magazine, not a journal. Which means that the article is being edited by an actual editor (Morgan Ryan, in this case). It's great: he read it over and made comments and changes, lots of things that make the article more readable and sensible.

It's fun having an editor. I've written about 250 articles and 6 books, but this is the first time I've ever had a real editor on anything I've written.

P.S. And, no, a copy editor is not the same thing.

Awhile ago I posted some maps based on the Pew pre-election polls to estimate how Obama and McCain did among different income groups, for all voters and for non-Hispanic whites alone. The next day the blogger and political activist Kos posted some criticisms. I disagree with one of Kos's suggestions--he wanted me to rely on exit polls, but I don't actually see them as more reliable than the Pew pre-election polls--but he pointed out some serious problems with my maps. I realized that some fixes were in order. Most importantly:

- My maps would be improved by replacing solid red and blue with continuous shading to distinguish between landslides and narrow margins.

- I needed a more flexible model that would allow the nonlinear pattern of voting and income to vary by state. (In the previous model, I fit a nonlinear pattern (by including a separate logistic regression coefficient for each of the five income categories) but allowed the states to vary only with intercepts and slopes. In the new model, we're letting all five coefficients vary by state.)

During the past couple of months, I've been working on this when I've had a spare hour or two, and now I think we have something reasonable to share. Here it is:

States colored deep red and deep blue indicate clear McCain and Obama wins; pink and light blue represent wins by narrower margins, with a continuous range of shades going to pure white for states estimated at exactly 50/50.

General comments

The maps are based on a model fit to four ethnic categories (non-Hispanic white, black, Hispanic, other), but I'm only displaying total and non-Hispanic whites. The others are interesting too but they're based on a lot less data: they're my (current) best estimates but are much more reliant on model extrapolation.

The estimates are entirely based on the Pew data--except that we use Census-based voter turnout estimates to reweight estimates in each state, and we shift each state's estimates to be consistent with the actual election outcome in the state. (For example, if our estimate says that Obama got 48% of the total vote in a state (adding up voters from all income and ethnicity categories), and he actually got 46%, then we'd pull down our estimates for each category so that the estimated total is 46%.)

Some particular changes

I'll talk about a couple of states where Kos pointed out issues with my original maps.

New Hampshire. John McCain won 45% of the two-party vote in New Hampshire, a state which is 93% non-Hispanic white, 1% black, 2% Hispanic, 2% Asian, and 2% other. Based on the Census survey, we estimate that non-Hispanic whites were 96% of New Hampshire's voters in 2008. If whites represented 96% of the voters, and if McCain received 20% of the votes of the other 4%, then his share of the white vote would be 46%--thus, as Kos pointed out, it's hard to believe that McCain won in four of the five income categories among whites in the state, as my original map had implied. The problem was in the way that I'd adjusted things to the national vote.

Michigan. As Kos points out, Michigan was closely divided among whites, and so there was something fishy about my original maps, which had Obama winning among whites in four of the five income categories. The new map does not have this problem.

Colorado. This state reveals some problems with the published exit poll data: according to CNN, McCain got 48% of the white vote in Colorado, but, when this was broken down by income, he got 45% of the vote of whites under $50,000 and 47% of the vote of whites over $50,000. This is a mathematical impossibility: using the exit poll numbers, McCain's percentage of the total white vote should then be (.19*45% + .62*47%)/(.19+.62) = 46.5%, not 48%. I don't know which of these--if either--is correct. I assume all of these numbers are from the corrected exit polls, adjusted to match up to the actual vote proportions in each state. Our estimate gives McCain 51% of the white vote in Colorado. I think this is possible too, and for that matter it's consistent with the exit poll estimate of 48%, which has a standard error of at least sqrt(.48*(1-.48)/(.81*1254))=.015, so the exit poll number is within two standard errors of our estimate.

Estimates and raw data

Here are graphs showing our estimates, along with the weighted average from the Pew surveys in each group.(including only those respondents who expressed a preference for Obama or McCain and also said they were "absolutely certain" they had registered to vote):

You can see the partial pooling from the data to the model, with more pooling in small states such as Wyoming, Rhode Island, and Vermont, and less pooling in states such as California, Texas, and New York where sample sizes were larger. The graphs show estimated McCain vote share, so, unsurprisingly, the lines for whites are higher than the lines for all voters, with differences smaller in states such as Wyoming or Vermont where there are very few nonwhite voters.

Some technical details

Even after restricting to respondents who are certain they are registered, the pre-election polls don't do a great job matching the population of voters. To correctly weight to voters (rather than to the general adult population), we used the 2008 Current Population Survey post-election supplement, which has information on voter turnout. We'll write a technical article describing exactly what we did, but the short version is that the CPS numbers are generally considered to be much more reliable than exit polls or pre-election polls for estimating turnout rates among different groups within a state. What we actually did was to use a multilevel model to smooth the CPS numbers using the latest population totals from the American Community Survey.

Yair also came up with a cool color scheme. Instead of going from deep red to deep blue through purple, we divided up the color scheme as follows: for proportions between 0 and .5, we used different shades of blue (deep blue, getting progressively lighter, toward white), then going from .5 to 1, we used deeper and deeper reds, starting with white, through light pink, to red. (Don't worry, I'll post the R code.) This worked much, much better than the purple schemes I was playing with before. More visual resolution, and a key benefit is that it's immediately clear which states are above and below the 50% threshold. Finally, I did a little trick of my own and used a square-root transformation (more specifically, if the estimate vote proportion for McCain is x, I defined z = 2*(x-.5), and then wored with sign(z)*sqrt(z)) to spread out the resolution near 0.5 and compress it near 0 and 1.

One other thing. The Pew organization sent me their raw data and posted them on the web for anyone to use. The exit polls still refuse to report anything but summaries. I don't see this refusal as a sign of confidence on their part. Please also read my earlier note for further discussion of the Pew and exit polls.

All this work is joint with Yair Ghitza.

Recently on Gothamist, there was a post about this site. It depicts subway ridership since 1905, as measured at each subway stop (by annual recorded entries).

I wish that the graphs were click-able to enlarge them; though it's fun to look at this way, it's tough to compare the graphs with that tiny size. You can zoom in slightly to display the station names, which is nice. It seems as though this is still a work in progress on some fronts, however: you can't zoom in or out too far or you lose the map altogether.

My quick take on the Souter replacement is that, with 59 Democratic senators and high popularity, Obama could nominate Pee Wee Herman to the Supreme Court and get him confirmed. But I'm no expert on this. The experts are my colleagues down the hall, John Kastellec, Jeff Lax, and Justin Phillips, who wrote this article on public opinion and senate confirmation of Supreme Court nominees. They find:

Greater public support strongly increases the probability that a senator will vote to approve a nominee, even after controlling for standard predictors of roll call voting. We also find that the impact of opinion varies with context: it has a greater effect on opposition party senators, on ideologically opposed senators, and for generally weak nominees.

This looks like a cool book.

One of the chapters is by John Hughes. Perhaps he'll talk about cinematography? I have no idea what he's been up to lately. There's also this chapter, which I hope is beautiful in its content if not in its appearance.

Jimmy found this amusing.

Responding to my question about graphing horse race results, Megan Pledger writes:

While waiting up late to snipe at an internet auction, I put together some simple data of a horse race and used ggplot to plot it. It's discrete time race data rather than continuous time and has very simple choice options for the horse. The graph is a starting point!

[The picture doesn't fully fit on the blog window here; right-click and select "view image" to see the whole thing.]

My reply: Very nice--thanks! I won't look a gift horse in the mouth . . . but if I were to be picky, I'd suggest making the tods smaller, the lines thinner, and the colors of the five horses more distinct. All these tricks should make the lines easier to follow. I'd also suggest gray rather than black for the connecting lines.

I think I'd also supplement it with a blown-up version of the last bit (from 80-100 on the x-axis), since that's where some interesting things are happening.

And here's the code:

Christian Robert, Nicolas Chopin, and Judith Rousseau wrote this article that will appear in Statistical Science with various discussions, including mine.

I hope those of you who are interested in the foundations of statistics will read this. Sometimes I feel like banging my head against a wall, in my frustration in trying to communicate with Bayesians who insist on framing problems in terms of the probability that theta=0 or other point hypotheses. I really feel that these people are trapped in a bad paradigm and, if they would just think things through based on first principles, they could make some progress. Anyway, here's what I wrote:

I actually own a copy of Harold Jeffreys's Theory of Probability but have only read small bits of it, most recently over a decade ago to confirm that, indeed, Jeffreys was not too proud to use a classical chi-squared p-value when he wanted to check the misfit of a model to data (Gelman, Meng, and Stern, 2006). I do, however, feel that it is important to understand where our probability models come from, and I welcome the opportunity to use the present article by Robert, Chopin, and Rousseau as a platform for further discussion of foundational issues.

In this brief discussion I will argue the following: (1) in thinking about prior distributions, we should go beyond Jeffreys's principles and move toward weakly informative priors; (2) it is natural for those of us who work in social and computational sciences to favor complex models, contra Jeffreys's preference for simplicity; and (3) a key generalization of Jeffreys's ideas is to explicitly include model checking in the process of data analysis

I just happened to run across this today. It's awesome.

Justin Wolfers writes:

Dick [Easterlin] was the first economist to start taking subjective well-being data seriously. While this sort of research is now pretty mainstream, I have to imagine that it took a fair bit of courage back in the early 1970's.

This was interesting to me: the idea that it would take courage to study a particular research topic. Especially something such as subjective well-being, which doesn't have any direct political connections. I mean, it's not like we're talking about the economic benefits of torture, or whatever. "Subjective well-being" seems pretty innocuous to me: whatever objections made it courageous to study this topic must have been intellectual and stylistic rather than political.

P.S. Back when I taught at Berkeley, I did get some flak for doing research on Bayesian statistics--some students told me that other faculty had told them not to take my course--but I wouldn't describe my decision to do work on that topic as "courageous." I think the atmosphere in economics in the 1970s must have been much different than anything I've ever experienced.

This is oddly compelling:

The color scheme is boring--it just replicates geographic information that is already clear from the picture. I'd prefer a more informative color scheme, perhaps based on per-capita GDP, but that's a minor quibble. (With the new color scheme, it might help to outline the continents in gray to make it easier to locate everything.)

Also, of course the dots are not necessary, but maybe they give the map some of its charm. Lower-case letters are certainly much easier to distinguish than upper-case letters.

One other point that otherwise might be missed: What really makes this map work is that it does not display the borders between the countries. Border displays draw attention to the countries' shapes, which is not usually what we care about. That's one reason why I'm not a fan of those distort-a-maps that stretch out states or countries in proportion to their population.

I received the following unsolicited email from Ashley.Dittmar@medqueryinc.com with subject line: "Invitation to consult on research regarding wound complications resulting from surgery." I'm sure that any of you could respond. See below: