November 2005 Archives

We finally finished the paper (and here's the earlier blog posting with link to a powerpoint).

I don't need art to be work-related. In fact, I generally prefer that it's not. But there's an exhibition at MOMA called SAFE: Design Takes On Risk, that looks pretty cool. Items range from practical (chairs with well-placed hooks to hide a purse) to pseudo-practical (suitcase-like containers to keep bananas from getting bruised) to borderline neurotic (slip-on fork covers). And earplugs. Lots of earplugs. For those who don't live in the city or just don't want to shell out the $20 entrance fee, there's an online exhibition: http://moma.org/exhibitions/2005/safe/.

I recently met Carlos Davidson, a prof at Cal State University. He studies amphibians, with a special interest in why frogs in California are disappearing. He said that he can "predict quite well whether a site will have frogs, based on the pesticide use upwind" and that he thinks that pesticides are a big part of the problem. But he also said that others in his field are far from convinced. What should it take to be convincing? Is there a "statistical" answer to questions like, which is more important: lab work, more field work, more analysis of existing field data (perhaps with more covariates included)?

Last week I substitute professed a mathematical statistics course for a friend who was out of town. I was sort of dreading it: interpretation of confidence intervals, Fisher information, AND hypothesis tests, all in one class, less than 24 hours before the start of Thanksgiving break. I didn't have high hopes for the enthusiasm level in the room. BUT it was actually pretty fun. The Cramer-Rao inequality? It's really cool that there's a derivable bound on the variance of an unbiased estimator, and even cooler that that bound happens to be the inverse of the Fisher information. It's not the kind of stuff that comes up much in my own work or that I'd want to do research on myself, but I got a kick out of teaching it.

During my visit to George Mason University, Bryan Caplan gave me a draft of his forthcoming book, "The logic of collective belief: the political economy of voter irrationality." The basic argument of the book goes as follows:

(1) It is rational for people to vote and to make their preferences based on their views of what is best for the country as a whole, not necessarily what they think will be best for themselves individually.
(2) The feedback between voting, policy, and economic outcomes is weak enough that there is no reason to suppose that voters will be motiaved to have "correct" views on the economy (in the sense of agreeing with the economics profession).
(3) As a result, democracy can lead to suboptimal outcomes--foolish policies resulting from foolish preferences of voters.
(4) In comparison, people have more motivation to be rational in their conomic decisions (when acting as consumers, producers, employers, etc). Thus it would be better to reduce the role of democracy and increase the role of the market in economic decision-making.

Caplan says a lot of things that make sense and puts them together in an interesting way. Poorly-informed voters are a big problem in democracy, and Caplan makes the compelling argument that this is not necessarily a problem that can be easily fixed--it may be fundamental to the system. His argument differs from that of Samuel Huntington and others who claimed in the 1970s that democracy was failing because there was too much political participation. As I recall, the "too much democracy" theorists of the 1970s saw a problem with expectations: basically, there is just no way for "City Hall" to be accountable to everyone, thus they preferred limiting things to a more manageable population of elites. Caplan thinks that voting itself (not just more elaborate demands for governmental attention) is the problem.

Bounding the arguments

I have a bunch of specific comments on the book but first want to bound its arguments a bit. First, Caplan focuses on economics, and specifically on economic issues that economists agree on. To the extent the economists disagree, the recommendations are less clear. For example, some economists prefer a strongly graduated income tax, others prefer a flat tax. Caplan would argue, I think, that tax rates in general should be lowered (since that would reduce the role of democratic government in the economic sphere) but it would still be up to Congress to decide the relative rates. This isn't a weakness of Caplan's argument; I'm just pointing out a limitation of its applicability.

More generally, non-economic issues--on which there is no general agrement by experts--spread into the economic sphere. Consider policies regarding national security, racial discrimination, and health care. Once again, I'm not saying that Caplan is wrong in his analysis of economic issues, just that democratic goverments do a lot of other things. (At one place he points out that the evidence shows that voters typically decide whom to vote for based on economic considerations. But, even thought the economy might be decisive on the margin, that doesn't mean these other issues don't matter.)

Finally, Caplan generally consideres democracy as if it were direct. But I think representative democracy is much different than direct democracy. Caplan makes some mention of this, the idea that politicians have some "slack" in decision-making, but I suspect he is understating the importance of the role of the politicians in the decision-making process.

Specific comments

"Weekend at Bernie's" is a low-quality movie that nobody's seen but everybody knows what it's about. Are there many other examples of this sort of cultural artifact? Another is Woody Allen's movie Zelig, which didn't get great reviews or great box office, but once again, its theme is well known. It's tough for me to think of lots of examples of this sort. The key is to have all 3 features:
(1) Generally acknowledged to be of low quality
(2) Not particularly popular or successful
(3) Basic storyline or theme is well known.

For example, James Joyce's Ulysses satisfies (2) and (3) but not (1); similarly (at a lower level) with Edward Scissorhands. The Bridges of Madison County satisfies (1) and (3) but not (2), and of course lots of things satisfy (1) and (2) but not (3). I also don't want to include "so-bad-it's-good" kinds of artifacts like "Plan 9 from Outer Space" which are famous because of their crappiness.

I'm really thinking of things like Weekend at Bernie's, which sought, and found a low-to-moderate success but had a storyline with such a good "hook" that most of the people who know about it didn't actually see the movie (or read the book, or whatever). It's not such a mystery that lots of people know what Ulysses is about--it's a Great Book so we've heard about it. And it's not such a mystery that lots of people know what The Bridges of Madison County is about--lots of people bought the book. But Weekend at Bernie's (or Zelig)--they probably have really good gimmicks to be so well known.

One of the people I met in my visit to George Mason University was Robin Hanson. At lunch we had a lively conversation about democracy--Hanson thinks it's overrated! When I (innocently) told him that representative democracy seemed better than the alternatives, he pointed out that there are some successful alternatives out there. For example, Microsoft is a (de-facto) dictatorship, and it does pretty well. Did I think Microsoft would work better as a democracy, he asked? Well, hmm, I don't know much about Microsoft, but yeah, I suppose that if some element of representative democracy were included, where employees, customers, and other "stakeholders" could vote for representatives that would have some vote in how things were run, maybe that would work well. At this point, someone else at lunch (sorry, I don't remember his name) objected and said that only shareholders should have the vote. I said maybe that's true about "should," but in terms of actual outcomes I wouldn't be surprised if things could be improved by including employees (and suppliers, customers, etc) in the decision-making. Bringing back to the main discussion, Robin pointed out that I had retreated from the claim that "democracy is best" to the claim that "some democracy could make things a little better." He said that his point about democracy's problems leads him to want to restrict the range of powers given to a democratic government and make the private sphere larger.

I don't really know what to say about that--the distinction I was making was between "pure democracy" and "representative democracy." My impression from the work in social and cognitive psychology on information aggregation is that representative democracy will work better than dictatorship or pure democracy.

Anyway, Robin gave me a copy of his paper proposing decision-making using betting markets. It's an interesting paper, sort of a mix of a policy proposal and a criticism of our current political system. Here's the abstract:

Democracies often fail to aggregate information, while speculative markets excel at this task. We consider a new form of governance, wherein voters would say what we want, but speculators would say how to get it. Elected representatives would oversee the after-the-fact measurement of national welfare, while market speculators would say which policies they expect to raise national welfare. Those who recommend policies that regressions suggest will raise GDP should be willing to endorse similar market advice. Using a qualitative engineering-style approach, we present three scenarios, consider thirty design issues, and then present a more specific design responding to those concerns.

It's an interesting paper and I have a few comments and questions:
- On page 8, Hanson notes that "betting markets beat major opinion polls." I think betting markets are great, but comparing to opinion polls is a little misleading--a poll is a snapshot, not a forecast (see here for elaboration on this point).
- On page 10, Hanson proposes using GDP as a measure of policy success. When I read this, I thought, why not just use some measure of "happiness," as measured in a poll, for example? One problem with a measure from a survey is that then the survey response itself becomes a political statement, so if, for example, you oppose the current government, you might be more likely to declare yourself "unhappy" for the purpose of such a poll. Joe Bafumi has found such patterns in self-reports of personal financial situations. GDP, on the other hand, can't be so easily manipulated. For the purposes of Robin's paper, I guess my point is that these properties of the "success measure" are potentially crucial.
- On page 11, Handon says, "an engineer [as compared to a scientist] is happy to work on a concept with a five percent chance of success, if the payoff from success would be thirty times the cost of trying." I would hope that a scientist would think that way too!
- On pages 11-12, he says that "scientists usually have little use for prototypes and their tests..." This may be true of some scientists, but "prototypes" (in the sense of data analyses that illustrate new or untested methods) play a huge role in statistics. In fact, this may characterize most of my own published papers!
- On page 12, Hanson writes that "most corporations are in effect small democratic governments." However, the vote of stockholders is not representative democracy as in U.S. politics, with defined districts, regularly scheduled elections for representatives, and so forth. I think this makes a big difference.

Now for my larger questions, which I think reflect my confusion about how this proposal would actually be implemented.

- Choice of policies to evaluate. I don't quite see how you would decide which potential policies get a chance of being evaluated in the prediction market. There could be potentally thousands or millions of policies to compare, right? On page 26, Hanson suggests limiting these via a $100,000 fee, but this would seem to shut a lot of people out of the system. (Of course, Hanson might reply that the current system, in which politicians from Bloomberg to Bush can parlay money into votes, also has this problem. And I would agree. I'm just trying to understand how the current system would work. In practice, would there need to be a system of "primary elections" or "satellite tournaments" to winnow the proposals?

- Picking which proposal to implement. Suppose two or more conflicting proposals are judged (by the prediction markets) to improve expected GDP. Which one would be implemented? This sort of problem would just get worse if there were thousants of proposals to compare.

In some ways, this reminds me my idea of "institutional decision analysis," which is that formal decision rules are appropriate for "institutional" settings where there is agreement on goals and also the need for careful justification of decisions. Similarly, Hanson's "futarchy" technocratically formalizes decisions that otherwise would have been made politically (though bargaining, persuasion, maneuvering, manipulation of rules, and so forth).

Sometimes people ask me to help them write more clearly. A common difficulty is that the way that first seems natural to write something, is not always the best way.

For example, I just (in an email) wrote the sentence, "The ultimate goal is to understand the causal effect on asthma of traveling to/from Puerto Rico." I started to write it as, "The ultimate goal is to understand the causal effect of traveling to/from Puerto Rico on asthma." This made more sense (the causal effect of X on Y), but it's more confusing to read (I think) because "on asthma" is at the end of the sentence, so when you get there you have to figure out where it goes. The sentence I actually wrote has the form, "the causal effect on Y of X," which is more awkward logically but easier to read.

P.S. Yes, I'm sure I've made at least one writing mistake in the above paragraph (otherwise I'd be violating Bierce's Law, and I wouldn't want to do that). But I think my main point is valid.

It's College Week at Slate: Click here for the thoughts of several prominent academics on improving undergraduate education, sometimes with the aid of a magic wand. I of course first read "Learn Statistics. Go Abroad" by K. Anthony Appiah. I completely agree with Dr. Appiah's view that many college graduates can't evaluate statistical arguments, leaving them unequipped to make informed decisions in areas such as public policy. He writes "So I favor making sure that someone teaches a bunch of really exciting courses, aimed at non-majors in the natural and social sciences, which display how mathematical modeling and statistical techniques can be used and abused in science and in discussions of public policy." Again, I agree completely. But (as we've discussed here and here) teaching those kinds of courses is really hard, and probably requires that magic wand.

Ben Cowling asks,

What do you think of the growing area of 'expert trading markets' using expert opinion for predicting future events (as compared to, say, formal mathematical or statistical models incorporating past data in the forecasting process)? From what I can gather the markets produce a form of informative prior so perhaps the whole process might be considered as a kind of simple mathematical model(?)

I'm motivated by the recent article in the economist:

Science and Technology: Trading in flu-tures; Predicting influenza
The Economist: 377 (8448) p. 108. Oct 15, 2005.

But I know these expert markets have been used in other areas; the Iowa Electronic Market is claimed to be good at predicting all sorts of things successfully including elections, which is why I thought you and readers of your blog might be interested.

My response: I first heard of the Iowa markets nearly 15 years ago, when Gary King and I were writing our paper about why pre-election polls vary so much when elections are so predictable. For this paper, all we needed to establish was that elections are predictable, which indeed they are, using state-by-state regression forecasting models (as was done in the 1980s and 1990s by Rosenstone and Campbell, and more recently by Erikson, among others). The Iowa markets also give good forecasts, which isn't a suprise given that the investors in these markets can use the regression forecasts that are out there.

Basically, my impression is that the prediction markets do a good job at making use of the information and analyses that are already out there--for elections, this includes polls and also the information such as economic indicators and past election results, which are used in good forecasting models. The market doesn't produce the forecast so much as it motivates investors to find the good forecasts that are already out there.

As an aside, people sometimes talk about a forecasting model, or a prediction market, "outperforming the polls." This is misleading, because a poll is a snapshot, not a forecast. It makes sense to use polls, even early polls, as an ingredient in a forecast (weighted appropriately, as estimated using linear regression, for example) but not to just use them raw.

P.S. In the comments, Chris points out this interesting article on prediction markets by Wolfers and Zitzewitz.

Here's an example of how the principles of statistical graphics can be relevant for displays that, at first glance, do not appear to be statistical. Below is a table, from a Language Log entry by Benjamin Zimmer, of instances of phrases of the form "He eats, drinks, sleeps X" (where the three verbs, along with X, can be altered). I'll present Zimmer's table and then give my comment.

Here's the table:

I agree completely with this Junk Charts entry, which presents two examples (from the Wall Street Journal on 9 Nov 2005) of bar graphs that become much much more readable when presented as line graphs. The trends are clearer, the comparisons are clearer, and the graphs themselves need much less explaining. Here's the first graph (and its improvement):

And here's the second:

My only (minor) comments are:

- First graph should be inflation-adjusted.
- Both graphs could cover a longer time span.
- Axis labels on second graph could be sparer (especially the x-axis, which could be labeled every 5 years).
- I'd think seriously about having the second graph go from 0 to 100% with shading for the three categories (as in Figure 10 on page 451 of this paper, which is one of my favorites, because we tell our story entirely through statistical graphics).
- The graphs could be black-and-white. I mean, color's fine but b&w is nice because it reproduces in all media. The lines are so clearly separated in each case that no shading or dotting would be needed.

P.S. See here for a link to some really pretty graphs.

Per Pettersson-Lidbom is presenting a paper (in the Political Economy Seminar) that claims that an increase in size of local-government legislatures decreases the size of local government. First I'll give his abstract, then my comments. The abstract:

This paper addresses the question of whether the size of the legislature matters for the size of government. Previous empirical studies have found a positive relationship between the number of legislators and government spending but those studies do not adequately address the concerns of endogeneity. In contrast, this paper uses variation in council size induced by statutory council size laws to estimate the causal effect of legislature size on government size. These laws create discontinuities in council size at certain known thresholds of an underlying continuous variable, which make it possible to generate “near experimental” causal estimates of the effect of council size on government size. In contrast to previous findings, I [Pettersson-Lidbom] find a negative relationship between council size and government size: on average, spending and revenues are decreased by roughly 0.5 percent for each additional council member.

It's cool how he uses a natural experiment based on the laws of Finland and Sweden. As he writes: "In Finland, the council size of local governments is determined solely by population size. For example, if a local government has a population between 4001 and 8,000, the council must consist of 27 members, but if its population is between 8,001 and 15,000 the council must have 35 members. Thus, the law creates a discontinuity in council size at the threshold of 8001 inhabitants." And regression-discontinuity analysis certainly seems appropriate here.

The actual result is suprising to me--not that I'm any expert on local government, it's just surprising to see a negative effect here--and so I'd like to see some presentation of the data. If the effect is as clear as is claimed, it should show up in some basic analyses--here I'm thinking of scatterplots and matching analyses. This is somewhat a matter of taste--as a statistician, I like graphs, but economists seem to prefer tables. But I just find it difficult to be convinced by results such as Tables 9-14.

To flip it around: this is a pretty clean dataset, right? You have a natural experiment and some points near the boundary. So a scatterplot, and a simple regression could be pretty convincing. Tables 2 and 3 are promising (well, I'd prefer graphs, but still...) but they only have data on "x", not on "y". As things stand, I really just have to take the results on trust. Not that I have any reason to disbelieve them, but I'd like to be a little more confident in the results--especially given that much of the paper discusses why these results differ from the rest of the literature on the topic.

Why do Supreme Court justices drift toward the center? This seems to have occured with some Republican-appointed justices over the past few decades and with some Democrat-appointed justices in earlier years. The natural comparison here is to Congress, where I don't know of any evidence of center-drifting or leftward-drifting of Congressmembers. My law-professor colleague explains the difference as coming from the enviornment of the court. In particular, each case gets discussed and argued (rather than simply voted upon, as with many bills in Congress).

He also points out that judges, by the nature of their job, are exposed to two sides of every issue. Over the years, this could tend to lead to moderation. Unfortunately, most of us do not generally have to seriously consider two sides of every issue. Once again, the comparison to Congress is instructive: Congressmembers are exposed to a lot of lobbyists, who are certainly not divided evenly on issues. (Just pick your favorite issue here: drugs, guns, Israel, . . .) My argument here is not that lobbyists have too much (or not enough) influence but rather that as a judge, you get exposed to arguments in a more structured and balanced way, which might lead to moderation.

Looking at the data

Aleks sent along the following graph of Supreme Court justices, for each year, plotting the proportion of cases for which each judge voted on the conservative side (as coded by Spaeth):

(See here for the bigger version.)

This is a pretty picture--I particularly like the careful use of colors (the original version had some dotted lines but I talked Aleks into just using solid lines)--but one has to be careful in its interpretation. In particular, the graph is telling us about the relative position of justices in any given year, but I wouldn't trust its implicit claims about changes from year to year, or its long-term trends. The difficulty is that the results shown in this graph depend on the case mix in any given year.

Much of the year-to-year variation in your graph can be attributed to variation in the docket. It's not clear how to make sense of long-term trends given that the docket is changing over time. In particular, I wouldn't be surprised if the docket has become more conservative in recent years--as the court has shifted, i'd expect the cases to shift also. Another example is Marshall and Brennan from 1970 to 1990. Do I really believe that they both got more liberal, then both got more conservative, then both more liberal? Well, maybe, but it seems more plausible to me that the docket was changing during these times. This is related to the problem in epidemiology of simultaneously estimating age, period, and cohort effects.

Here's Kevin Quinn's estimate of ideal points of Supreme Court justices:

Clearly we need to combine Kevin's modeling tools with Aleks's graphics! (Joe, David, Noah, and I fit our own Supreme Court model, but I'm embarrassed to say it didn't allow judges' ideologies to move over time. And here's Simon Jackman's overview of ideal-point models.)

Tyler Cowen links to an article from 1999 by Mark Greenberg that discusses the so-called "doomsday argument," which holds that there is there is a high probability that humanity will be extinct (or drastically reduce in population) soon, because if this were not true--if, for example, humanity were to continue with 10 billion people or so for the next few thousand years--then each of us would be among the first people to exist, and that's highly unlikely.

Anyway, the (sociologically) interesting thing about this argument is that it's been presented as Bayesian (see here, for example) but it's actually not a Bayesian analysis at all! The "doomsday argument" is actually a classical frequentist confidence interval. Averaging over all members of the group under consideration, 95% of these confidence intervals will contain the true value. Thus, if we go back and apply the doomsday argument to thousands of past data sets, its 95% intervals should indeed have 95% coverage. If you look carefully at classical statistical theory, you'll see that it makes claims about averages, not about particular cases.

However, this does not mean that there is a 95% chance that any particular interval will contain the true value. Especially not in this situation, where we have additional subject-matter knowledge. That's where Bayesian statistics (or, short of that, some humility about applying frequentist inferences to particular cases) comes in. The doomsday argument is pretty silly (and also, it's not Bayesian). Although maybe it's a good thing that Bayesian inference has such high prestige now that it's being misapplied in silly ways. That's a true sign of acceptance of a scientific method.

Sy Spilerman writes,

I am interested in the effect of log(family wealth) on some dependent variable, but I have negative and zero wealth values. I could add a constant to family wealth so that all values are positive. But I think that families with zero and negative values may behave differently from positive wealth families. Suppose I do the following: Decompose family wealth into three variables: positive wealth, zero wealth, and negative wealth, as follows:

- positive wealth coded as ln(wealth) where family wealth is positive, and 0 otherwise,
- zero wealth coded 1 if the family has zero wealth, 0 otherwise.
- negative wealth coded ln(absolute value of wealth) if family wealth is negative, and 0 otherwise,

and then use this coding as right side variables in a regression. It seems to me that this coding would permit me to obtain the separate effects of these three household statuses on my dependent variable (e.g., educational attainment of offspring). Do you see a problem with this coding? A better suggestion?

My reply:

Yes, you could do it this way. I think then you'd want to include values very close to zero (for example, anything less than $100 or maybe $1000 in absolute value) as zero. But yes, this should work ok. Another option is to just completely discretize it, into 10 categories, say.

Any other suggestions out there? This problem arises occasionally, and I've seen some methods that seem very silly to me (for example, addiing a constant to all the data and then taking logs). Obviously the best choice of method will depend on details of the application, but it is good to have some general advice too.

Mark Liberman replied to this entry (see also here):

I [Mark Lieberman] was mostly trying to see whether a new database search program was working. I knew that men have been said to use filled pauses like "uh" more than women, and it made sense to me that disfluency would increase with age, so I generated the data for the first plot and took a look. I think you're right that I should have started the plot from 0, but I wasn't sure what I'd see, and thought that the qualitative effects if any would be clearer with a narrower range of values plotted.

Then I wondered about "um", and still had a few minutes, so I ginned up the data for the second plot and took a look at it. I was quite surprised to see the opposite age effect, and somewhat surprised to see the inverted sex effect, so I quickly looked up the standard papers on the subject and banged out a post.

Actually what I did was to add a bit of verbiage around the .html notes (with embedded graphs) that I'd been making for myself.

I've attached the first plot that I made in that session, showing the female/male ratio for a number of words that I thought might show a difference. The X axis is the (log) count of the word (mean of counts for male and female speakers), and the y axis is the (log) ratio of female/male counts. The plotted words are too small, but I wasn't sure how much they would overlap...

If I can find another spare hour or two, I'm going to check out whether southerners really talk slower than northeners.

And here's Mark's new plot:

Here's the full version. (I don't know how to fit it all on the blog page.)

P.S. In his new plots (see here), Mark uses a 2x2 grid and extends the y-axis to 0. To be really picky, I'd suggest making 0 a "hard boundary." In R you can do this using 'yaxs="i"' in the plot() call, but then the top boundary will be "hard" also, so that you have to use ylim to extend the range (e.g., ylim=c(0,1.05*max(y))). What I should really do is write a few R functions to encode my default graphing preferences so that I don't need to do this crap every time I make a graph.

Mark Liberman posted some interesting summaries of telephone speech records from the Linguistic Data Consortium. He writes:

I [Mark Liberman] took a quick look at demographic variation in the frequency of the filled pauses conventionally written as "uh" and "um". For technical reasons that I won't go into here, I used the frequency of the definite article "the" as the basis for comparison. Thus I selected a group of speakers (e.g. men aged 60-69), counted how often they were transcribed as saying "uh", and to normalize that count (since the number of people in each category was different) I divided by the number of times the same speakers were transcribed as saying "the".

He also did "Um":

My comments

And now, some contentless comments about graphical presentation:

1. I like the clear axis labels and titles, and even more importantly, that the lines are labeled directly (rather than using different dotted lines and a key). Good labeling is important--I do it even for the little graphs I'm making in my own research when exploring data or model fits.
2. I would've used blue for boys and pink for girls--easier to remember--although perhaps Mark was purposely trying to be non-stereotypical.
3. My biggest change would have been to (a) put the 2 graphs on a common scale, and (b) make them smaller, and put them next to each other. Smaller graphs allow us to see more at once, and see patterns that can be more obscure when we are forced to scroll back and forth between mutiple plots. In R, I do par(mfrow=c(2,2)) as a default.
4. I would have the bottom of each graph go to 0, since that's a natural baseline (the zero-uh and zero-um level that we might all like to try to reach!). There's been some debate about the "start-at-zero rule" but I usually favor it in a situation such as this, where it doesn't require much extension of the axis.

Anyway, Mark's blog entry has much more on this interesting data source.

P.S.

Caroline says "emmm" instead of "ummm." Is this standard among native Spanish speakers?

P.P.S.

See here and here for more.

There's been some debate in the media and among social scientists about the relation between income and voting. On one hand, the states that support the Democrats--the so-called "blue states"--are richer, on average, than the Republican-leaning "red states." On the other hand, richer voters continue to support the Republicans--not so much as an economic determinst might suspect (even in the lowest income category, Bush in 2004 still got 36% of the vote) but the correlation is there.

Awhile ago, we made this plot, which shows how the Republicans can simultaneously have the support of poor states, and richer voters within states:

Mississippi is the poorest state, Ohio is in the middle, and Connecticut is the richest state. Within each state, the line shows the probability of supporting Bush for President for each of five income categories, and the five open circles represent the relative proportion of adults in that state in each category. The black circles show the average income and probability of supporting Bush for each state.

The above plot was fit with a model (a varying-intercept logistic regression) that restricted the slopes in the states to be essentially parallel. We then expanded the model to allow the slopes to vary also, so that the coefficient for income could differ in richer or poorer states. The figure below shows the result:

Income clearly matters much more in "red states" like Mississippi than in "blue states" like Connecticut. We also see this pattern in 2004, and somewhat in 1992 and 1996, but not really before the 90s.

There's an article by Stephen Dubner and Steven Levitt in the New York Times today on why it's rational to vote. They correctly point out that the probability of casting a decisive vote is very small. Unfortunately they don't seem to be aware of the social-benefit motivation for voting, which is why voting is, in fact, rational behavior. See here for our paper on the topic. For convenience, I'll repeat our earlier blog entry below. The short version is that you can be rational without being selfish.

The chance that your vote will be decisive in the Presidential election is, at best, about 1 in 10 million. So why vote?

In another example of the paradox of importance, a colleague writes:

In other news, I am about to use the "hot deck" method to do some imputation. I considered using one of the more sophisticated and generally better methods instead, but hey, I'm on a deadline, plus there are many other sources of error that will be larger than the ones I'm introducing. It's the same old story/justification for using linear models, normal models, assuming iid errors, etc.

At least he feels bad about it. That's a start.

My colleague Jan Vecer in the statistics department at Columbia gave a talk the other day on "Crash options." His claim was that the introduction of such options could have a socially beneficial effect by allowing investors to plan more effectively in the context of market instabilities. I'm in no position to evaluate this one way or another, but it sounded like a cool idea, so I'm passing it along.

Here's Jan's abstract:

In this paper, we introduce new types of options which do not yet exist in the market, but they have some very desirable properties. These proposed contracts can directly insure events such as a market crash or a market rally. Although the currently traded options can to some extent address situations of extreme market movements, there is no contract whose payo® would be directly linked to the market crash and priced and hedged accordingly as an option.

Here's the paper, and here are the slides from a talk he gave on the topic.

Unfortunately, his paper has no cool graphs. I've suggested to Jan that he make a graph to show how the crash option could work to stabilize the market. I know he has the ability to make cool graphs; see his paper on tiebreakers in tennis and here for an article about his tennis predictor.

Statistics is fundamental to pharmacology and drug development. Billy Amzal at Novartis forwarded me this job announcement for a statistician or mathematician who wants to do statistical modeling in pharmocokinetics/pharmacodynamics. "Knowledge of Bayesian statistics and its application is a strong plus." It's a long way from Berkeley, where one of my colleagues told me that "we don't believe in models" and another characterized a nonlinear differential equation model (in pharmacokinetics) as a "hierarchical linear model." Anyway, it looks like an interesting job opportunity.

Somebody saw this entry and called, saying that he heard that I was going to Barcelona.

As every statistician knows, many people hate our field. How many times have we all heard "You do statistics? I HATED that class in college!" (I remember one of my college professors complaining indignantly that no one would presume to tell an artist that he hated art.) There are all sorts of factors that probably contribute to the unpopularity of statistics: it's often one of the few quantitative courses required for social science majors, who may be less into mathy subjects to begin with; it's not always well-taught (although what subject is?); the logic of hypothesis testing isn't terribly intuitive. Lately I've been wondering if statistics' bad reputation is confined to the US or if it's more universal. My own experiences really don't help to answer that question: Sure, most of the "I hate statistics" comments I hear come from Americans, but that's not surprising given that I live in the US. And many of the international people I know are work-related, so of course they tend not to hate statistics. Anyway, my wondering about this is self-serving. Starting in January I'll be teaching intro statistics at Pompeu Fabra University in Barcelona, and I've been wondering what the students will be like. I'm hoping not to hear "me disgusta estadística" too much....

Jim Greiner has an interesting note on the use of statistics in racial discrimination cases. As both a lawyer and a statistician, Jim has a more complete perspective on these issues than most people have. I won't comment on the substance of Jim's comments (basically, he claims that the statistical analyses in these cases, on both sides, are so crude that judges can pretty much ignore the quantitative evidence when making their decisions) since I know nothing about the case in question. But I do have a technical point, which in fact has nothing really to do with racial discrimination and everything to do with statistical hypothesis testing.

Jim writes,

The facts of the specific case, which concerned the potential use of race in preemptory challenges in a death penalty trial, are less important than Judge Alito's approach to statistics and the burden of proof.

Schematically, the facts of the case follow this pattern: Party A has the burden of proof on an issue concerning race. Party A produces some numbers that look funny, meaning instinctively unlikely in a race-neutral world, but conducts no significance test or other formal statistical analysis. The opposing side, Party B, doesn't respond at all, or if it does respond, it simply points out that a million different factors could explain the funny-looking numbers. Party B does not attempt to show that such innocent factors actually do explain the observed numbers, just that they could, and that Party A has failed to eliminate all such alternative explanations.

. . .

Is there a middle way? Perhaps. In the above situation, what about requiring some sort of significance test from Party A, but not one that eliminates alternative explanations? In the specific facts of Riley, the number-crunching necessary for "some sort of significance test" is the statistical equivalent of riding a tricycle: a two-by-two hypergeometric with row totals of 71 whites and 8 blacks, column totals of 31 strikes and 48 non-strikes, and an observed value of 8 black strikes yields a p-value of 0.

OK, now my little technical comment. I don't think the hypergeometric distribution is appropriate since it conditoins on both margins. The relevant margin to condition on is the number of whites and blacks, since that was determined before the lawyers got to the problem. In a hypothesis-testing framework in which p-values represent the probability of various hypothetical alternatives (this is the framework I like, it can be interpreted classically or Bayesianly). To put it another way, the so-called Fisher exact test isn't really "exact" at all.

This is just a rant I go on occasionally, really has nothing to do with Jim's note except that it reminded me of the issue. For the fuller version of this argument, see Section 3.3 of my paper on Bayesian goodness-of-fit testing in the International Statistical Review. Also, Jasjeet Sekhon wrote a paper recently on the same topic.

For Jim's specific example, I'd be happy just doing a chi-squared test with 1 degree of freedom. His calculation is fine too--the hypergeometric is a reasonable approximation to a Bayesian posterior p-value with noninformative prior distribution.

P.S.

See also this item in Chance News.

Here's the abstract to today's brown bag seminar in the Marketing Department (331 Uris Hall, 1:30pm, for you locals). If you read the abstract you'll see the Halloween connection.

On the Consumption of Negative Feelings
(Eduardo B. Andrade, UC Berkeley & Joel B. Cohen, University of Florida)

Abstract:

If the hedonistic assumption (i.e., people’s willingness to pursue pleasure and avoid pain) holds, why do individuals expose themselves to events known to elicit negative feelings? In this article, we assess how (1) the intensity of the negative feelings, (2) the positive feelings in the aftermath, and (3) the coactivation of positive and negative feelings contribute to our understanding of the phenomenon. In a series of 4 studies, horror and non-horror movie watchers are asked to report their positive and negative feelings either after (experiment 1) or while (experiments 2A, 2B, and 3) they are exposed to a horror movie. The results converge with a coactivation-based model and highlight the importance of a protective frame.