June 2009 Archives

At our sister blog, Tom Schaller says no:

Is Sanford a cad for bolting his family on Father's Day weekend? Of course, but that is a private, moral failing, rather than a failure of public duty. . . .

I [Schaller] oppose most of what Mr. Sanford stands for politically. His showy rejection of federal stimulus money targeted for his state was a crass publicity stunt designed to garner national attention for Mr. Sanford at the expense of his constituents, many of whom are struggling economically. . . . Should Mr. Sanford's ambitions founder on the shoals of a personal scandal, however, yet another opportunity will be lost to establish the long-overdue separation between private comportment and public service. So here's hoping he doesn't resign or, if he does, it is a matter of personal choice rather than him bowing to political pressure.

I see where Schaller is coming from. Lots of people have complicated personal lives, and it's not clear at all that these difficulties have much if anything to do with governing. But I don't know if I agree with him on the wall of separation between private comportment and public service.

Consider the Sanford case. Schaller's a Democrat, so he can evaluate Sanford on his policies. But if Schaller were a Republican, he might very well want Sanford out of there because he tarnishes the brand, makes the party a laughingstock, etc. Also makes it harder for Sanford to convincingly follow a "family values" agenda which Schaller (if he were a Republican) might want. These are legitimate concerns for a Republican to have. Even if you don't think Sanford's personal indiscretions are important, you might want him gone and replaced by a more effective Republican. Just as, from the other direction, a Democrat would've preferred a zipped-fly version of Bill Clinton.

Some time ago FlowingData had an article on visualizing tables - which really is about visualizing spreadsheets in terms of correlations between columns. While Circos generates very colorful displays:

Today I was impressed by a much cleaner and Tuftier variant on the theme by Mike Bostock, called Dependency Tree:

Click on the link, it's interactive. Jeff Heer and Bostock also have a new JavaScript visualization toolkit out ProtoVis, which simplifies the creation of such stuff. The computer scientist in me finds this development very cool. But I still like my correlation matrices.

Sometimes you hear discussion of how the red states get more from the government than they pay in taxes while the blue states get less and pay more. This is slightly misleading because the blue states are richer and rich people pay a higher rate of income tax, but it does raise the interesting question of the regionally distributive effects of national taxing and spending poliicies.

For some perspective on where this is coming from: In our office is a map from 1924 titled "Good Roads Everywhere" that shows a proposed system of highways spanning the country, "to be built and forever maintained by the United States Government." The map, made by the National Highways Association, also includes the following explanation for the proposed funding system: "Such a system of National Highways will be paid for out of general taxation. The 9 rich densely populated northeastern States will pay over 50 per cent of the cost. They can afford to, as they will gain the most. Over 40 per cent will be paid for by the great wealthy cities of the Nation. . . . The farming regions of the West, Mississippi Valley, Southwest and South will pay less than 10 per cent of the cost and get 90 per cent of the mileage." Beyond its quaint slogans ("A paved United States in our day") and ideas that time has passed by ("Highway airports"), the map gives a sense of the potential for federal taxing and spending to transfer money between states and regions.

P.S. Yes, I posted this last year, but without the pretty map image (click on it for higher resolution, which unfortunately still isn't quite good enough to make out the text)..

The Howard Wainer story.

On of the fun parts is this story from his days as an assistant professor:

Back in April, in an article about partisan perceptions of the economy, John Sides and I wrote:

A colleague and I were talking the other day about how much we pay our research assistants. It turns out that she pays much more. In fact, sometimes I don't get around to paying my research assistants at all, but she pays hers a decent amount.

My colleague, who's an untentured professor, said that was understandable because she makes less money than I do, so she can better relate to the students' lifestyles. That's a pretty scary thought--it should really go the other way, right? I get paid more so I should be able to afford to be more generous. But maybe she's right; if so, it's a sobering insight.

One major impediment, scientists agree, is the grant system itself. It has become a sort of jobs program, a way to keep research laboratories going year after year . . .

I was on an NIH panel a couple of years ago with about 25 other scientists, reviewing something like 90 grants. It was pointless. 25 people is just too many to make a decision. What happened was that there were 3 or 4 people who were experienced in the process, who ended up guiding the entire discussion.

The highlight--or, I should say, lowlight--was when we were reviewing a proposal involving the study of the carcinogenic effects of hookah (water pipe) smoking. I asked if this was really such a big deal, and one of the panel members told me that smoking tobacco through a hookah is something like 10 times worse than smoking a cigarette. If so, the public health consequences could be pretty serious, even if not so many people did it. I said this sounded like a reasonable point to me. Then this guy across the table from me spoke up and said that he knew somebody who was 80 years old, had been smoking with a hookah all his life and was none the worse from it. At this point, I blew up. I couldn't believe that the "my elderly aunt smokes and she didn't get cancer" argument could be brought up at an NIH panel!

My final thoughts on those Iran vote analyses:

Our article (by Yu-Sung, Jennifer, Masanao, and myself, and based also on work with Kobi, Grazia, and Peter Messeri) will be appearing in the Journal of Statistical Software, in a special issue on missing-data imputation. Here's the abstract:

Our mi package in R has several features that allow the user to get inside the imputation process and evaluate the reasonableness of the resulting models and imputations. These features include: flexible choice of predictors, models, and transformations for chained imputation models; binned residual plots for checking the fit of the conditional distributions used for imputation; and plots for comparing the distributions of observed and imputed data in one and two dimensions. In addition, we use Bayesian models and weakly informative prior distributions to construct more stable estimates of imputation models. Our goal is to have a demonstration package that (a) avoids many of the practical problems that arise with existing multivariate imputation programs, and (b) demonstrates state-of-the-art diagnostics that can be applied more generally and can be incorporated into the software of others.

We've made lots of improvements since listing the package last year (here). There's still a lot more work to do, in many different directions (including multilevel models, nonignorable models, the self-cleaning oven, and making the program run faster in sorts of ways), and we keep improving it. But it's good to have something out there.

To actually get the R package, just open your R window, click on Packages, Install packages, and grab mi.

Pinchas Lev writes:

Sometimes people think it's a disaster when you have more predictors than data points, but I always point out that, no, it's better to have 9 predictors than just 1 or 2. After all, if you really wanted just 1 or 2, you could just throw out most of your data!

Nate's chart is excellent, especially the ordering of the candidates in order of the percent favoring resignation:

I also like the gratuitious exclamation marks which add fun value without actually making the graph any harder to read. The key reason this works is that Nate wisely did not fill in the blank squares with "No!"s.

My only comments are:

Andrew Knight points me to this Kafkaesque report on Bayesian methods and evidence-based medicine. It's always good to see things like this out there,

My main disagreement with the report is on their framework in which there is a fixed data model and different choices of prior distribution. As we discuss in Section 2.8 of Bayesian Data Analysis, I much prefer the framework in which a single prior distribution (or "population distribution") is applied to many different data settings. I think that framing it my way makes the benefits of Bayesian inference much clearer.

I also don't like all the tables. But that's not really a Bayesian issue.

I've been assured, and I believe, that the effective way to get rid of the roaches in your apartment is to clean the place, put poison in the cracks, and then seal them. Some people do that. But a lot of people go for the "bombing" approach: the exterminator comes to the building once a month, drops the bomb, leaves, and comes back the next month.

My question is: what are these people thinking?? Why do these people willingly get bombed once a month instead of following the simpler and effective approach? Part of this is ignorance, surely, but I think there's more to it than that, some underlying psychological appeal. I don't think it's just ignorance because, when I talk with people who get bombed and discuss the "clean, poison, and seal" approach, I've found them to be very resistant and (I would say) "defensive." They seem to want to believe that bombing is effective and really don't want to hear about alternative strategies.

What's going on? I have some theories. Maybe bombing seems like less effort than cleaning the food out of your closet and sealing the cracks. Also it seems sort of decisive. On the other hand, shouldn't people pause a little when they think about needing the exterminator every month? Yet, that doesn't seem to bother people. Conceptually, getting the exterminator to bomb your apartment feels to me a bit like "taking a pill." Maybe there's some technological appeal. Sort of like the way that photovoltaics are sexy in a way that passive solar isn't.

I don't know. I'll have to ask some psychologists of my acquaintance who work on environmental decision making.

Hall, J.L., L.W. Miratrix, P.B. Stark, M. Briones, E. Ginnold, F. Oakley, M. Peaden, G. Pellerin, T. Stanionis and T. Webber, 2009. Implementing Risk-Limiting Audits in California, USENIX EVT/WOTE, In press.

Related discussion here.

Donna Harrington writes:

I will be teaching a new multilevel models course in the fall and am currently reading your text, /Data Analysis Using Regression and Multilevel/Hierarchical Models/ as I prepare. I am enjoying the book and am considering adopting it for use in the course.

Would you be willing to share the syllabus you have used for your Applied Regression and Multilevel Models course? I am particularly interested in seeing how much of the book you use in a one semester course.

My reply:

I have to admit that, over the years, I've made my syllabuses less and less detailed as I've focused more and more on writing the books. For a multilevel modeling course, I suggested the following:

- chapters 3,4,5: linear and logistic regression
- chapter 7: basics of simulation
- chapter 9: basics of causal inference
- chapters 11-14: multilevel linear and logistic regression (up to and including varying-intercept, varying-slope models)
- chapter 18: all the theory that they'll need.

For a one-semester introductory course, my usual strategy for a one-semester course is to focus chapters 2-10: that is, cover everything except multilevel modeling. Linear regression, logistic, glm, computation, and causal inference. Then for the last part of the course, I can choose among some options, including: intro to multilevel models, sample size and power calculations, and missing data imputation.

P.S. To those of you who haven't had the opportunity to take a course from me: Don't worry about it. I'm better at writing than teaching. Maybe you're better off learning out of one of my books with somebody else actually teaching the class.

From H. Russell Bernard and Chris McCarty.

A political scientist writes:

Here's a question that occurred to me that others may also have. I imagine "Mister P" will become a popular technique to circumvent sample size limitations and create state-level data for various public opinion variables. Just wondering: are there any reasons why one wouldn't want to use such estimates as a state-level outcome variable? In particular, does the dependence between observations caused by borrowing strength in the multilevel model violate the independence assumptions of standard statistical models? Lax and Phillips use "Mister P" state-level estimates as a predictor, but I'm not sure if someone has used them as an outcome or whether it would be appropriate to do so

First off, I love that the email to me was headed, "mister p question." And I know Jeff will appreciate that too. We had many discussions about what to call the method.

To get back to the question at hand: yes, I think it should be ok to use estimates from Mister P as predictor or outcome variables in a subsequent analysis. In either case, it could be viewed as an approximation to a full model that incorporates your regression of interest, along with the Mr. P adjustments.

I imagine, though, that there are settings where you could get the wrong answer by using the Mr. P estimates as predictors or as outcomes. One way I could imagine things going wrong is through varying sample sizes. Estimates will get pooled more in the states with fewer respondents, and I could see this causing a problem. For a simple example, imagine a setting with a weak signal, lots of noise, and no state-level predictors. Then you'd "discover" that small states are all near the average, and large states are more variable.

Another way a problem could arise, perhaps, is if you have a state-level predictor that is not statistically significant but still induces a correlation. With the partial pooling, you'll see a stronger relation with the predictor in the Mr. P estimates than in the raw data, and if you pipe this through to a regression analysis, I could imagine you could see statistical significance when it's not really there.

I think there's an article to be written on this.

Robin Hanson writes,

In academia, one often finds folks who are much more (or less) smart and insightful than their colleagues, where most who know them agree with this assessment. Since academia is primarily an institution for credentialling folks as intellectually impressive, so that others can affiliate with them, one might wonder how such mis-rankings can persist.

I added the bold font myself for emphasis. Granted, Robin is far from a typical economist. Nonetheless, that he would write such an extreme statement without even feeling the need to justify it (and, no, I don't think it's true, at least not in the "academia" that I know about) . . . that I see as a product of being in an economics department.

P.S. Robin definitely is correct about the "more (or less) smart and insightful" bit. But here I think there are two things going on. First, in any group of people you'll see some variation, especially given that there are other factors going on than "smart and insightful" when it comes to selecting people in an academic environment. Second, there's more to life--even to academic life--than being smart and insightful. Even setting aside teaching, advising, administration, etc., some other crucial qualities for academic research include working hard, having the "taste" to work on important problems, intellectual honesty, and caring enough about getting the right answer. I know some very smart and insightful people who have not made the contributions that they are capable of, because (I think) of gaps in some of these other important traits.

My former Berkeley colleague Phil Stark has written a series of articles on election auditing which might be of interest to some of you. Here they are:

Stark, P.B., 2009. Auditing a collection of races simultaneously. Working draft.

Miratrix, L.W., and Stark, P.B., 2009. Election Audits using aTrinomial Bound. Submitted to IEEE Transactions on Information Forensics and Security: Special Issue on Electronic Voting.

Stark, P.B., 2009. Risk-limiting post-election audits: P-values from common probability inequalities. Submitted to IEEE Transactions on Information Forensics and Security: Special Issue on Electronic Voting.

Stark, P.B., 2009. CAST: Canvass Audits by Sampling and Testing. Submitted to IEEE Transactions on Information Forensics and Security: Special Issue on Electronic Voting.

Stark, P.B., 2008. A Sharper Discrepancy Measure for Post-Election Audits. Annals of Applied Statistics, 2, 982-985.

Stark, P.B., 2008. Conservative Statistical Post Election Audits. Annals of Applied Statistics, 2, 550-581.

Phil has an interesting background: he got into statistics after working on inverse problems in geology. The methods he uses are based on exact error bounds, really much different than the Bayesian stuff I do, much more focused on getting conservative p-values and the like. As a result, the things he does in his papers are nothing at all like what I would do in these problems.

In a larger sense, though, I believe in methodological pluralism, and I'm glad to see a researcher such as Phil, who's working from such a different statistical framework as mine, work on these problems.

See here for discussion.

Update here and data here. I haven't looked at this in detail, but Walter Mebane is the expert on this stuff so I'm inclined to believe him. Even though he uses tables instead of graphs.

Again, just to emphasize: this sort of statistical analysis doesn't prove anything by itself, but it can be useful in giving people a sense of where to focus attention if they want to look further.

How will support for same-sex marriage change over time? One way to speculate is to break down current support across age groups, and that's what Justin and I have done, building off of our forthcoming paper.

We plot explicit support for allowing same-sex marriage broken down by state and by age. Seven states cross the 50% mark overall as of our current estimates, but the generation gap is huge. If policy were set by state-by-state majorities of those 65 or older, none would allow same-sex marriage. If policy were set by those under 30, only 12 states would not allow-same-sex marriage.

Our article has appeared in The American Scientist. (Here's a link to the full article; hit control-plus to make the font more readable.) I highly recommend it for your introductory (or advanced) statistics classes. We start with a silly story of a flawed statistical analysis of sex ratios that managed to sneak into a serious scientific journal, then discuss general issues of how to interpret inconclusive statistical findings (including a brief analysis of data from People Magazine's 50 Most Beautiful People lists), and then loop back and discuss the statistical reasons that exaggerated claims can get amplified by the news media.

The article begins as follows:

Alex Scacco and Bernd Beber follow up on their analysis of the Iran election data:

After we wrote our op-ed using the province-level data, we've now also done some preliminary tests with the county-level data. In the latter dataset, the last digits don't appear fraudulent. Why might we find suspicious last digits at the province level, while, at the same time, Walter Mebane and Boudewijn Roukema find evidence that first and second digits are fishy at the county level?

We can only speculate about what happened behind closed doors, but here is a scenario of top-down fraud that is consistent with the patterns found in the quantitative analyses mentioned above:

Greg Mankiw writes:

The next time you hear someone cavalierly point to international comparisons in life expectancy as evidence against the U.S. healthcare system, you should be ready to explain how schlocky that argument really is.

He points to the following claim by Gary Becker:

National differences in life expectancies are a highly imperfect indicator of the effectiveness of health delivery systems.for example, life styles are important contributors to health, and the US fares poorly on many life style indicators, such as incidence of overweight and obese men, women, and teenagers. To get around such problems, some analysts compare not life expectancies but survival rates from different diseases. The US health system tends to look pretty good on these comparisons.

Becker cites a study that finds that the U.S. does better than Europe in cancer survival rates and in the availability of hip and knee replacements and cataract surgery.

It makes a lot of sense to think of health as multidimensional, so that some countries can do better in life expectancy while others do better in hip replacements and cancer survival.

But I disagree with Mankiw's claim that it's "schlocky" to compare life expectancy. If the U.S. really is spending lots more per person on health care and really getting less in life expectancy compared to other countries . . . that seems like relevant information.

Bernd Beber and Alex Scacco present another quantitative analysis of the Iranian election data, this time looking at last digits. They write:

[Suspicions of fraud] have led experts to speculate that the election results released by Iran's Ministry of the Interior had been altered behind closed doors. But we don't have to rely on suggestive evidence alone. We can use statistics more systematically to show that this is likely what happened. Here's how.

We'll concentrate on vote counts -- the number of votes received by different candidates in different provinces -- and in particular the last and second-to-last digits of these numbers. For example, if a candidate received 14,579 votes in a province (Mr. Karroubi's actual vote count in Isfahan), we'll focus on digits 7 and 9.

I want to explore the distinction between self-experimentation and formal experimentation in the context of a recent discussion on Seth's blog.

The story begins with two people who found, via self-experimentation, how to make their acne go away:

A student . . . had gone on a camping trip and found that her acne went away. At first she thought it was the sunshine; but then, by self-experimentation, she discovered that the crucial change was that she had stopped using soap to wash her face.

A friend of Seth writes: "I started "washing" my face with water about a month ago, and [now] my face is acne free and soft as a pair of brand new UGG boots. [He had had acne for years.]"

In the comments section, someone writes:

While it would be nice to think that all we have to do to get rid of acne is stop using those expensive cleanser and just use water - this is just anecdotal evidence you present. It would require a large clinical trial to be conclusive.

Seth replies that informal experimentation is cheaper and faster than more formal clinical trials. Also, different things might work for different people, so whether or not a treatment has been evaluated a large study, it might make sense to test it yourself--especially for something such as acne or weight loss that is not an urgent concern.

This got me thinking . . . what are the benefits (if any) of a formal controlled trial? In statistics, we usually frame these benefits by comparing to observational studies. The big risk in an observational study is that the treatment and control groups will differ in important ways (as in the famous hormone replacement therapy story). Is this worth the cost? Maybe. Sometimes.

A related issue is bias, a word which I am using in the conversational rather than the statistical sense. For example, how would you want to evaluate the risks and effectiveness of a new drug that was developed by a pharmaceutical company at the cost of millions of dollars? I'd be suspicious of an observational study: even if conducted by professionals, there just seem to be too many ways for things to be biased.

In Seth's acne example, there is no financial source of bias. And, as Seth points out, the test is free to apply on yourself. If I had a kid with acne, I'd give it a try and do an experiment--which means trying the soap and no-soap conditions on different days (or different weeks, or months) and measuring and recording acne levels. One thing I've gathered from Seth's work is that there are big benefits to be gained by doing self-experimentation with careful measurement and record keeping, rather than simply trying different things and trying to remember what works.

On the other hand, yeah, I'm skeptical about Seth's acne claims, and I think a larger study would be more likely to convince me. But I don't think it would have to be expensive. All Seth (or somebody) needs is to set up a protocol for deciding when to wash with soap or water and a protocol for measuring acne, then he could get a bunch of volunteers to flip coins and try it. This blog has a few thousand readers, and Seth's diet forum has thousands of participants, so it shouldn't be so hard to find people to do this. I'm not so interested in acne myself, but according to Seth (and others, I assume), "acne really matters," so maybe it's worth giving this a try.

This is not meant as a put-down of Roth; I think Marquand is great.

P.S. Is this what a Twitter post is like? Basically, I'm too lazy to back up my statement here with evidence. But I think that any of you out there who've read both Roth and Marquand will agree upon a moment's reflection that I'm right.

P.P.S. When I'm talking about Marquand, I exclude the way overrated The Late George Apley. I'm talking about real Marquand books like Point of No Return, Wickford Point, etc.

Good airplane reading. Lived up to the reviews.

The American Statistical Association awarded the 2009 Excellence in Statistical Reporting Award to Sharon Begley of Newsweek. From the official announcement:

The above remark, which came in the midst of my discussion of an analysis of Iranian voting data, illustrates a gap--nay, a gulf--in understanding between statisticians and (many) nonstatisticians, one of whom commented :that my quote "makes it sound that [I] have not a shred of a clue what a p-value is."

Perhaps it's worth a few sentences of explanation.

Benford's law is an amusing mathematical pattern in which the first digits of randomly sampled numbers tend to have a distribution in which 1 is the most common first digit, followed by 2, then 3, and so forth. It's the distribution of digits that arises from numbers that are sampled uniformly on a logarithmic scale.

In our Teaching Statistics book, Deb and I describe a classroom demonstration where we show how Benford's law applies to street addresses sampled randomly from the telephone book. In a more serious vein, Walter Mebane has written about the application of Benford's law to vote counts.

In the past several days, a few people have asked me about applying these ideas to the recent Iranian election. Today, Stephane Reissfelder pointed me to an article by Boudewijn Roukema, which states:

The results of the 2009 Iranian presidential election presented by the Iranian Ministry of the Interior (MOI) are analysed based on Benford's Law and an empirical variant of Benford's Law. The null hypothesis that the vote count distributions satisfy these distributions is rejected at a significance of p < 0.007, based on the presence of 41 vote counts for candidate K that start with the digit 7, compared to an expected 21.2-22 occurrences expected for the null hypothesis. A less significant anomaly suggested by Benford's Law could be interpreted as an overestimate of candidate A's total vote count by several million votes. Possible signs of further anomalies are that the logarithmic vote count distributions of A, R, and K are positively skewed by 4.6, 5.8, and 2.5 standard errors in the skewness respectively, i.e. they are inconsistent with a log-normal distribution with p ` 4 × 10−6, 7 × 10−9, and 1.2 × 10−2 respectively. M's distribution is not significantly skewed.

I don't buy it. First off, the whole first-digit-of-7 thing seems irrelevant to me. Second, the sample size is huge, so a p-value of 0.007 isn't so impressive. After all, we wouldn't expect the model to really be true with actual votes. It's just a model! Finally, I don't see why we should be expecting distributions to be lognormal.

Maybe there's something I'm missing here, but that's my quick take. This is not to say that I think the election was fair, or rigged, or whatever--I have absolutely zero knowledge on that matter--just that I don't find this analysis convincing of anything. I will say, though, that Roukema deserves credit for presenting the analysis clearly.

P.S. In response to comments: let me emphasize that I'm not saying that I think nothing funny was going on in the election. As I wrote, I'm commenting on the statistics, I don't know the facts on the ground. To move my comments in a more constructive direction (I hope), let me pull out this useful comment from Roukema's article: "One possible method to test whether this is just an odd fluke would be
to check the validity of the vote counts for candidate K in the voting areas
where the official number of votes for K starts with the digit 7." Further investigation could be a good thing here.

I did not find Roukema's argument convincing; that does not mean that I consider it a bad thing that the article was written. The article is a first draft of an analysis; it might end up leading to nothing, or it might be unconvincing as it stands now but lead to some important breakthroughs. We can see what further analysis turns up. Again, my verdict is not a Yes or a No, it's an "I'm not convinced."

From Flat Earth News:

You could argue that every profession has its defining value. For carpenters, it might be accuracy: a carpenter who isn't accurate shouldn't be a carpenter. For diplomats, it might be loyalty: they can lie and spy and cheat and pull all sorts of dirty tricks, and as long as they are loyal to their government, they are doing their job. For journalists, the defining value is honesty--the attempt to tell the truth. That is our primary purpose. All that we do--all that is said about us--must flow from the single source of truth-telling.

What is the defining value of statisticians?

P.S. My favorite of the responses below is Mike Anderson's:

Separate the signal from the noise, then look at the noise for more signals.

I like this because (a) it acknowledges the presence of "noise" (that is, variation) but (b) recognizes that the "signal" is what's most important.

Taxonomies are fractal with, at any node, some number of branches (typically one or two major branches and several minor ones). Here's a fascinating article by Benoit Mandelbrot from 1955 on models of taxonomic structures. Great stuff. The article was published in Information Theory--3rd London Symposium, ed. Colin Cherry, and is hard to find online. At least it was until now.

As part of our Red State, Blue State research, we developed statistical tools for estimating public opinion among subsets of the population. Recently Yu-Sung Su, Yair Ghitza, and I applied these methods to see where school vouchers are more or less popular.

We started with the 2000 National Annenberg Election Survey, which had responses from about 50,000 randomly-sampled Americans to the question: "Give tax credits or vouchers to help parents send their children to private schools--should the federal government do this or not?" 45% of those who expressed an opinion on this question said yes, but the percentage varied a lot by state, income level, and religious/ethnic group; These maps show our estimates:

(Click on image to see larger version.)

Vouchers are most popular among high-income white Catholics and Evangelicals and low-income Hispanics. In general, among white groups, the higher the income, the more popular are school vouchers. But among nonwhites, it goes the other way, with vouchers being popular in the lower income categories but then becoming less popular among the middle class.

You can also see that support for vouchers roughly matches Republican voting, but not completely. Vouchers are popular in the heavily Catholic Northeast and California, less so in many of the mostly Protestant states in the Southeast. We also see a regional pattern among African Americans, where vouchers are most popular outside the South.

We checked our results by fitting the same model to the Annenberg survey from 2004, and, much to our relief, we found similar patterns:

The only thing that puzzles me about this article (sent to me by Chris Wiggins) is that at first it's presented as new: "The trend is buried deep in United States census data . . " A couple paragraphs down, the article explains that these patterns were published last year by Lena Edlund and Doug Almond (who presented the results in our quantitative political science seminar). In any case, it's an excellent news article and discusses the issues well. The only thing I'd like to see are some sample sizes, so that students who are given this article to read can compute the standard errors on their own.

Also, I have a couple problems with their graph. First, I'm not a fan of expressing sex ratios as #boys per 100 girls. To me, it's clearer just to give %girls (or %boys) as a straight number: 48.8% or whatever. Second, it's a mistake to make these as bar graphs starting at zero. Here, zero is not a reasonable baseline: it's not like you're really expecting to see zero girl births. I appreciate that they were trying to make a pretty graph, but in this case I'd go with a simple dot plot with +/- 1 standard error bars on the points. Or, better still, a line plot with time on the x-axis (one point for each decade) lines connecting the dots for each ethnic group, and also the vertical lines indicating standard errors.

Line plots are the best, and it's great when you can put time on the x-axis.

"A fondness for collecting a salary and getting away with as little intellectual intercourse as possible is endemic to the academic world." Not just the academic world, I think. Working is hard work. That's why they call it work. On the other hand, I'm doing this for free.

This issue reminds me of a discussion that's sometimes come up about a well-known listserv participant who is (a) very helpful, and (b) very rude. Or maybe I'm exaggerating a bit: this person is (a) often helpful, and (b) often rude. Anyway, I've always maintained that, rudeness aside, this person is altruistic, providing free statistical help to strangers. But it's true that answering listserv questions isn't intellectually taxing. Sort of like writing this blog, it's work-like without usually quite being work.

P.S. I think the point is best made by keeping the listserv and its well-known participant anonymous.

Fred Bookstein was at my talk in Seattle on voting power (the relevant articles are here and here) but didn't get a chance to ask a question, so he's asking it now:

Why is voting power considered a "good" in all those models? What is good about it? With what generally shared human desiderata, if any, is it associated?

As the saying goes, everybody wants to go to heaven but nobody wants to die. Or, to put in political terms, people want lower taxes and more government services--with the gap filled, presumably, with a mixture of borrowed funds and savings realized by cutting government waste. In their new book "Class War? What Americans Really Think about Economic Inequality," Benjamin Page and Lawrence Jacobs put together survey data and make a convincing case that this cynical story is not a fair summary of public opinion in the United States. Actually, most Americans--Democrats and Republicans alike--support government intervention in health care, education, and jobs, and are willing to pay more in taxes for these benefits.

Page and Jacobs recognize that Americans are confused on some of these issues, for example not realizing that sales taxes cost lower-income people more, as a percentage of their earnings, while the personal income tax hits higher-income groups more, on average. The result is widespread confusion about what are the most effective ways to pay for government spending. People are also confused about how to cut the budget. To choose a well-known example that is not in the book at hand, Americans overwhelmingly support reducing the share of the federal budget that goes to foreign aid, but they also vastly overestimate the current share of the budget that goes to this purpose (average estimate of 15%, compared to an actual value of 0.3%).

Confusions on specific tax and budget items aside, Page and Jacobs are persuasive that majority public opinion is consistent with tax increases targeted to specific government programs aimed at bringing a basic standard of living and economic opportunity to all Americans. They discuss how survey respondents generally feel that such an expansion of the role of government is consistent with generally expressed free-market attitudes, a philosophy which they call "conservative egalitarianism."

This is a book of public opinion, not policy, and the authors offer no judgment on whether the public's majority preference is achievable. For example, a vast majority of Americans--including 80% of Republicans--feel that "Government should spend whatever is necessary to ensure that all children have really good public schools they can go to" (p. 59), and another clear majority--this time including 60% of Republicans--agree with the statement that "The government in Washington ought to see to it that everyone who wants to work can find a job" (p. 62). It is an open question whether these goals are possible given the tax increases that voters are willing to accept.

Carl Klarner writes:

I'm currently doing work on state legislative elections that uses Democratic success as the dependent variable. I do these analyses with either the percent of the two-party vote for the Democrat as Y, or a dichotomous measure of a Democratic victory as Y.

Jeff Lax and Justin Phillips posted this summary of attitudes on a bunch of gay rights questions:

They did it all using multilevel regression and poststratification. And a ton of effort.

P.S. My only criticisms of the above graph are:

(a) I'd just put labels at 20%, 30%, 40%, etc. I think the labels at 25, 35, etc., are overkill and make the numbers harder to read. And the tick marks should be smaller.

(b) The use of color and the legend on the upper left are well done. But they should place the items in the legend in the same order as the averages in the graphs. Thus, it should be same-sex marriage, then 2nd parent acdoption, then civil unions, then health benefits, and so forth.

As I noted a couple days ago, gay marriage has had the largest recent increases in popularity in liberal states where the general population was already pro-gay.

But if you count the number of same-sex couples, you see something different, with the fastest increases in conservative areas of the country. Gary Gates writes:

You discussed the issue of social networks and knowing gay people as a possible explanation. You might want to look at some of the work of Greg Herek (a psychologist at UC-Davis) who is now saying that "knowing" someone is becoming a much less salient predictor of support for gay rights. Since nearly everyone now knows a gay person, he claims that the issue today is more whether or not you have a closer personal relationship with a gay person.

Your findings were also intriguing to me when comparing them to some of the work I [Gates] have done on the enumeration of same-sex couples in the US Census and the American Community Survey. This paper looks at changes in the counts over time.

I [Gates] find the largest changes (which I interpret as increased visibility of same-sex couples) in the most conservative parts of the country.

I looked at Gates's report and it looks like good stuff. It would definitely be a good idea to reconcile his findings of the largest increases in conservative parts of the country, with Lax and Phillips's findings that public opinion on gay marriage has changed fastest in liberal states.

I saw this one today, can't figure it out:

"Don't take away my rights because you won't control your child"

What is this, the right to punch somebody else's kids?? I can't imagine somebody exercising that particular right very often before getting hurt.

It's a funny thing: we typically think of bumper sticker slogans as being simplistic, but in this case it appears to be the opposite: the compression of an idea into a short phrase has made it incomprehensible to outsiders such as myself. Or maybe that's the point. I wouldn't want to see the owner of this car near any kids, that's for sure.

David VandenBos writes:

I stumbled upon your blog a few weeks ago . . . However, a good amount of your technical articles go over my head because of my lack of statistics education/training/experience. Do you have any basic reading suggestions for learning applied statistics? My organization captures tons of info and safely tucks it away into databases, but I'm really interested in learning how to get it out and make use of it.

Does anybody have any suggestions? I like my book with Jennifer but maybe there's something more basic to start with? There's also this online book on statistical graphics by Rafe Donahue which is actually fun to read.

P.S. I don't think any of the usual intro stat books would be good here. I think they focus too much on conventional topics and not enough on applied statistics. Not really the fault of these books: they're designed for the undergraduate curriculum, not for practitioners.

Kieran points me to this.

Fancy statistical analysis can indeed lead to better understanding.

Jeff Lax and Justin Phillips used the method of multilevel regression and poststratification ("Mister P"; see here and here) to estimate attitudes toward gay rights in the states. They put together a dataset using national opinion polls from 1994 through 2009 and analyzed several different opinion questions on gay rights.

Policy on gay rights in the U.S. is mostly set at the state level, and Lax and Phillips's main substantive finding is that state policies are strongly responsive to public opinion. However, in some areas, policies are lagging behind opinion somewhat.

A fascinating trend

Here I'll focus on the coolest thing Lax and Phillips found, which is a graph of state-by-state trends in public support for gay marriage. In the past fifteen years, gay marriage has increased in popularity in all fifty states. No news there, but what was a surprise to me is where the largest changes have occurred. The popularity of gay marriage has increased fastest in the states where gay rights were already relatively popular in the 1990s.

In 1995, support for gay marriage exceeded 30% in only six states: New York, Rhode Island, Connecticut, Massachusetts, California, and Vermont. In these states, support for gay marriage has increased by an average of almost 20 percentage points. In contrast, support has increased by less than 10 percentage points in the six states that in 1995 were most anti-gay-marriage--Utah, Oklahoma, Alabama, Mississippi, Arkansas, and Idaho.

Here's the picture showing all 50 states:

I was stunned when I saw this picture. I generally expect to see uniform swing, or maybe even some "regression to the mean," with the lowest values increasing the most and the highest values declining, relative to the average. But that's not what's happening at all. What's going on?

Some possible explanations:

- A "tipping point": As gay rights become more accepted in a state, more gay people come out of the closet. And once straight people realize how many of their friends and relatives are gay, they're more likely to be supportive of gay rights. Recall that the average American knows something like 700 people. So if 5% of your friends and acquaintances are gay, that's 35 people you know--if they come out and let you know they're gay. Even accounting for variation in social networks--some people know 100 gay people, others may only know 10--there's the real potential for increased awareness leading to increased acceptance.

Conversely, in states where gay rights are highly unpopular, gay people will be slower to reveal themselves, and thus the knowing-and-accepting process will go slower.

- The role of politics: As gay rights become more popular in "blue states" such as New York, Massachusetts, California, etc., it becomes more in the interest of liberal politicians to push the issue (consider Governor David Paterson's recent efforts in New York). Conversely, in states where gay marriage is highly unpopular, it's in the interest of social conservatives to bring the issue to the forefront of public discussion. So the general public is likely to get the liberal spin on gay rights in liberal states and the conservative spin in conservative states. Perhaps this could help explain the divergence.

Where do we go next in studying this?

- We can look at other issues, not just on gay rights, to see where this sort of divergence occurs, and where we see the more expected uniform swing or regression-to-the-mean patterns.

- For the gay rights questions, we can break up the analysis by demographic factors--in particular, religion and age--to see where opinions are changing the fastest.

- To study the "tipping point" model, we could look at survey data on "Do you know any gay people?" and "How many gay people do you know?" over time and by state.

- To study the role of politics, we could gather data on the involvement of state politicians and political groups on gay issues.

I'm sure there are lots of other good ideas we haven't thought of.

P.S. More here.

Fred Bookstein writes:

Your blog comment about triple-blinding was a joke, but there IS a triple-blinding procedure in which the identity of the two groups is not revealed to the statistician on the project until the very end. At all times the data analyses proceed solely in reference to a comparison of some unspecified "group A" with a similarly unspecified "group B," and the identification of who were the intervened-upon and who were not is concealed from him or her until the computations are finished. (There are some other assumptions, e.g. absence of baseline differences, required for this to make sense; it applies mainly in contexts like randomized clinical trials.) You can't really purge the Discussion section of an article of the possibility of spin, but at least you can get the right scatters and tables into the dossier that they're spinning. The possibility was called to my attention a while ago by Michael Myslobodsky, a wise old man from my schizophrenia research world, who did not remotely intend it as a joke.

Interesting. My only experience along these lines is when I was working with a student doing matching for a public health study: There were something like 100 treated units and 1000 potential controls, and we wanted to select 300 of these as matched controls. The researchers were careful to give us only the background information and no outcomes.

Google just launched a pre-alpha "Fusion Tables". The visualization capability is okay, the interface is not fully stable, but the cool thing is the ability to merge two tables, something I've spent a lot of time doing manually in the past, or with ad-hoc scripts.

Here's an example where I merge their GDP table with a disease table. I need to pick the "WHO Regions/Country" in the right column, so that both tables get aligned:

Afterwards, I can do a scatter plot of GDP rank (X) with child mortality/1000 (Y):

So, high GDP makes child mortality less likely, but not always, and it's not a correlation.

Even if Fusion tables is pre-alpha, the table fusion capability makes it immediately useful. The collaboration features look cool, but it will take some time to get them to work right. Then we'll have proper horizontal collaboration.

Mark Bucciarelli writes:

I'm interesting in applying the Bayesian ANOVA you described in your 2005 paper to some data I am analyzing.

Is your arm package for R the place to start? (I'm on Linux/Mac, so I'll have to build OpenBugs and maybe update the RBugs package.)

Or is there a more direct path?

I'm analyzing the impact of ad attributes on the variance in click rates; e.g., product category, time of day, graphical vs. text, etc, etc.

My reply: For most of the computations in that article, I actually used a Fortran program that I wrote (and ran from Splus). There's no way this could be recovered; it would be easier to start from scratch. For the last example (in Figures 6 and 7), I used Bugs; actually, I repeated this example in my book with Hill. For your example, I'd suggest Bugs if your sample size isn't too large. Or you could try Doug Bates's lmer() function which we use in our arm package. lmer doesn't currently express uncertainty in the variance parameters but it's a good start, certainly much better than trying to use R's aov() function or similar procedures in other packages.

A correspondent who prefers to remain anonymous asks:

Since you publish a lot of papers, I wonder if you've ever come across this issue. Journal reviews are supposed to be double-blind, but authors always have great familiarity with their own work, and cite it frequently. So what is the sense of sending an "anonymized" review copy to a journal editor when a line like "In a previous paper (Smith and Jones, 1999) we showed that ..." lets you know right away that Smith and Jones are the authors of the paper being reviewed?

I have thought about altering the review copy to make it look we are citing a paper by someone else ["In a previous paper, Smith and Jones (1999) showed that..."]. Should I even worry about this? How do you handle it?

My reply: I don't think it matters much. If the rules say to anonymize the references, then I do so, but I don't really worry whether a reviewer can figure out whether it is me writing the article. From the other direction, I review lots of articles (more than I write, actually), and I am very rarely curious enough to bother trying to figure out (for example, using Google) who is writing them.

What bothers me more, actually, is the idea that somebody out there is submitting a crappy article but citing me in such a way that the reviewers think I wrote it. The other thing I worry about is when I review an article negatively, that the authors might be able to figure out that I'm the reviewer. Or, that someone else is reviewing an article negatively and in the review points to my work, leading the author to think that I'm being the bad guy.

P.S. Somebody once told me about triple-blind submission, where even the author doesn't know who wrote the article. Apparently this is standard in medical research.

P.P.S. More thoughts here.

I used to display my .png files using the default viewer in Windows. Then Aleks told me about Irfanview. It's much better.

As I've discussed here on occasion, I like to standardize continuous regression inputs by dividing by two standard deviations. That way the rescaled variables each have sd of 1/2, which is approximately the same sd as any binary predictor, allowing the coefficients to be interpreted together.

Standardizing is often thought of as a stupid sort of low-rent statistical technique, beneath the attention of "real" statisticians and econometricians, but I actually like it, and I think this 2 sd thing is pretty cool.

As Aleks pointed out, however, standardizing based on the data is not strictly Bayesian, because the interpretation of the model parameters then depends on the sample data. As we discussed, a more fully Bayesian approach would be to think of the scale for standardization as an unknown parameter to itself be estimated from the data.

P.S. Recall that "inputs" are not the same as "predictors."

P.P.S. I scale by 2 sd to be consistent with 0/1 predictors. In retrospect, I wish I'd scaled by 1 sd and then coded binary predictors as -1 and 1 to be consistent. This would've been simpler overall. But I think it's too late now.

(Right click on "View image" to see the whole thing.)

Among whites, vouchers appear to be more popular with the upper middle class and rich (with predictable religious variation: the strongest support is among Catholics, then born-again Protestants, then others).

Among blacks and hispanics, though, vouchers are more popular among the poor.

We'll have to check this on some other data.

Some details:

You can recognize celery in an x-ray scanner.

The other day while waiting for a bus, I was thinking about how city buses should be smaller and run more frequently. Instead of a 40-seater every 15 minutes, they could run a 10-seater every 5 minutes. (More precisely, they could run as frequently as necessary during rush hour to handle all the passengers--a bus a minute if necessary--but more spaced out at other times. For example, on weekend mornings the bus is never crowded, so they could run the much smaller buses with just slightly higher frequencies than they currently run big buses now.)

The advantages of my proposal are clear: the bus comes more frequently, also since the lag time is smaller, loading and unloading won't take so much time, and as an extra bonus, you'll probably skip a lot more stops because there are fewer people on the bus who might want to get off at any particular point. Also, I don't know about fuel efficiency, but I wouldn't be surprised if the fuel cost per passenger is lower because you're not having to run these huge empty buses in off-peak hours. Finally, van-sized buses could maneuver better in traffic.

The only additional cost that I see is having to hire more bus drivers, but with unemployment at 9%, I don't think it would be hard to find people to do this. What really irritates me are those huge, huge buses that take forever to fill up and take about a half hour just to go a few crosstown blocks. If they were broken up into vans, the wait would be less and the ride much more pleasant.

P.S. Yes, I know this isn't one of the world's most important problems. But it is a big expenditure, so why not try to do it right?

P.P.S. I'm sure there's lots of research on this topic but it's not something I'm at all informed on. The above are just my personal impressions.

P.P.P.S. To those of you who discuss the cost: Sure, it would cost money. But there's a real economic benefit: people would be able to get around the city faster! A good use of stimulus funds, etc.

I am at a conference which had an excellent poster session yesterday. I realized the session would have been even better if the students with posters had been randomly assigned to stand next to and explain other students' posters. Some of the benefits:

1. The process of reading a poster and learning about its material would be more fun if it was a collaborative effort with the presenter.

2. If you know that someone else will be presenting your poster, you'll be motivated to make the poster more clear.

3. When presenting somebody else's poster, you'll learn the material. As the saying goes, the best way to learn a subject is to teach it.

4. The random assignment will lead to more inderdisciplinary understanding and, ultimately, collaboration.

I think just about all poster sessions should be done this way.

P.S. In reply to comments:

- David writes that my idea "misses the potential benefit to the owner of the poster of geting critical responses to their work." The solution: instead of complete randoimization, randoimize the poster presenteres into pairs, then put pairs next to each other. Student A can explain poster B, student B can explain poster A, and spectators can give their suggestions to the poster preparers.

- Mike writes that "one strong motivation for presenters is the opportunity to stand in front of you (and other members of the evaluation committee) and explain *their* work to you. Personally." Sure, but I don't think it's bad if instead they're explaining somebody else's work. If I were a student, I think I'd enjoy explaining my tellow-students' work to an outsider. The ensuing conversation might even result in some useful new ideas.

- Lawrence suggests that "the logic of your post apply to conference papers, too." Maybe so.

At our sister blog, Lee discusses strategic retirement, or lack thereof, in the Supreme Court.

This is a good time for me to bring up my point that congressmembers and senators appear to decide make these decisions non-strategically, being more likely to retire when their party most needs them and their incumbency advantage and being less likely to retire when the could be replaced more costlessly. (For example, Frank Lautenberg running for reelection in 2008, a year when the Democrats could well have afforded a fresh face in a New Jersey senate race with little chance of losing.)

A couple days ago, I wrote, of Martin and Quinn's estimated positions of Supreme Court justices, that

I don't know whether to believe the numbers. Is the Anthony Kennedy of 2007 (ideology score 0.14) really so close to Hugo Black in 1970 (ideology score 0.06)? To look at it another way, according to these numbers, in 1973 (the year of Roe v. Wade), six of the justices are colored red and the median justice is listed at 0.67. In 2007, only five are red and the median is at 0.14. In fact, in 2005 the median is listed as -0.07, or slightly to the left of center. Is it really plausible that the court was more liberal in 2005 than in 1973? Maybe so, but something looks fishy to me here.

In reply, Andrew Martin wrote:

re: Black and Kennedy, I [Martin] tend to think of them as pretty similar. Both were moderates (although on somewhat different types of cases), one a moderate Dem the other a moderate Rep. So them being close is not implausible.

There were a couple of very liberal decisions in the 1972 term (when Roe was decided), including a Roe and a death penalty case. But even on civil liberties the court reached a conservative decision in Miller (the obscenity case). And there were some more conservative decisions in other areas of law. Today's court is surely more conservative on civil liberties issues (although there haven't really been many cases...), but may be a little to the left on some other issues (Hamdan). Today it all gets down to what Kennedy wants to do. If it were what Roberts would do the court would be far to the right.

That's an argument for plausibility, but the argument may be implausible. We tend to think about the Court in terms of the most politically salient cases, but the model treats Roe as equally to, say, a tax case. And, of course, the measures have huge limitations because they are just based on binary data, on all cases, with some reasonably strong model assumptions, etc.

The point about counting different domains of the law is interesting, along with the age-period-cohort sort of question of how you can even try to align left-right today with the corresponding positions in 1972.

Greg Mankiw looked up the Consumer Reports of ratings of car companies and found:

Dead last was Chrysler. CU recommended zero percent of the Chrysler vehicles they tested. That's right--zero. Second to last was General Motors. CU recommended 17 percent of GM models. By contrast, most other companies had half or more of their models get the thumbs up. Honda was the top ranked brand; CU recommended 95 percent of its models.

Mankiw writes:

Is it any surprise that Chrysler and GM are now in the process of going out of business? From the perspective of the Consumer Reports advice, it looks like their business model was to count on the ignorance of the buying public about the quality of their products. Their bankruptcy should perhaps be viewed as a success of the market system.

This makes sense to me, but I wonder if it explains too much. Presumably these companies have been making crappy cars for awhile. How did the companies stay alive so long? In all seriousness, perhaps the market system would've been more successful had it shut down those companies 10 or 15 years ago.

Beyond this is the principal-agent problem, or moral hazard, or whatever it's called, by which the people who make the decisions to make crappy cars are probably not actually going broke themselves: the companies might fall apart, but they'll do OK, I assume. So I can see how the companies could stay alive for awhile, living off their assets and their ability to borrow money. I just don't completely see it as a "success of the market" that they've been hanging on so long when the low quality of their products has been public knowledge.

The Numbers Guy has an article titled This U.K. Sheriff Cites Officials for Serious Statistical Violations, and a corresponding blog post:

Mobilized by distressingly low levels of public trust in official statistics, the U.K. government is embarking on a daring, and possibly unique, experiment. With broad support, Parliament in 2007 approved the formation of the U.K. Statistics Authority, a group with the budget, authority and independence to question other government agencies on the numbers they release to the public. [...]

The agency's task is a delicate one. If it uncovers reams of faulty data that might have been used in crafting public policy, Britons' fraying faith in public institutions could be further eroded.

Interesting, a truth-assurance agency would be a good thing, also useful for validating the truthfulness of other statements that often get twisted by marketing. We might be finally making progress with the problems that Josiah Stamp identified many years ago.

Pretty pictures (even if I don't believe the numbers being plotted); follow the link here.

1. Coalitions, voting power, and political instability.

Thurs 4 Jun, 3:30pm, Kane Hall 210 at the University of Washington. Part of the Math Across Campus series.

We shall consider two topics involving coalitions and voting. Each topic involves open questions both in mathematics (probability theory) and in political science.
(1) Individuals in a committee or election can increase their voting power by forming coalitions. This behavior yields a prisoner's dilemma, in which a subset of voters can increase their power, while reducing average voting power for the electorate as a whole. This is an unusual form of the prisoner's dilemma in that cooperation is the selfish act that hurts the larger group. The result should be an ever-changing pattern of coalitions, thus implying a potential theoretical explanation for political instability.
(2) In an electoral system with fixed coalition structure (such as the U.S. Electoral College, the United Nations, or the European Union), people in diferent states will have different voting power. We discuss some flawed models for voting power that have been used in the past, and consider the challenges of setting up more reasonable mathematical models involving stochastic processes on trees or networks.

2. Culture wars, voting and polarization: divisions and unities in modern American politics.

Fri 5 Jun, 9:45am, Kane Hall 225 at the University of Washington. Part of the 10th anniversary celebration of the Center for Statistics and the Social Sciences.

On the night of the 2000 presidential election, Americans sat riveted in front of their televisions as polling results divided the nation's map into red and blue states. Since then the color divide has become a symbol of a culture war that thrives on stereotypes--pickup-driving red-state Republicans who vote based on God, guns, and gays; and elitist, latte-sipping blue-state Democrats who are woefully out of touch with heartland values. But how does this fit into other ideas about America being divided between the haves and the have-nots? Is political polarization real, or is the real concern the perception of polarization? We address these questions using a results from our own research and that of others.

3. Creating structured and flexible models: some open problems.

Mon 8 Jun, 11am, Fairmont Lounge, St. John's College, 2111 Lower Mall, University of British Columbia. Statistics Department seminar.

A challenge in statistics is to construct models that are structured enough to be able to learn from data but not be so strong as to overwhelm the data. We introduce the concept of "weakly informative priors" which contain important information but less than may be available for the given problem at hand. We also discuss some related problems in developing general models for taxonomies and deep interactions. We consider how these ideas apply to problems in social science and public health. If you don't walk out of this talk a Bayesian, I'll eat my hat.

4. Red state, blue state, rich state, poor state.

Mon 8 Jun, 3pm, Fairmont Lounge, St. John's College, 2111 Lower Mall, University of British Columbia. Statistics Department seminar.

If you come to any of these, please ask lots of questions!

P.S. I've never spoken at UBC, but I have given a couple of talks in the statistics department at UW. The first time was twenty years ago. The talk went OK, I think--it was on medical imaging--but I did a horrible thing by leading off with a joke. I could probably get away with that now, but it didn't go over well then. In my defense, the joke was related to the topic of the talk. But it was a pretty bad joke. The second talk was about twelve years ago. The topic was model checking in spatial statistics. I think it went fine, but I recall that there was one spatial statistics expert in the audience who was disappointed at how simple my model was. It worked ok for what we were doing, though.

Drew Conway writes:

To paraphrase Bill James, the alternative to doing statistics is not "not doing statistics," it's "doing bad statistics."

Some people bemoan the excessive quantitative nature of academic political science nowadays. I certainly agree that there's room for nonquantitative work, but you also want to have some people who know their way around numbers. Or else you'll end up with this sort of horrible non-analysis by David Runciman of U.S. elections. What's striking about Runciman's article--and he's a well-respected political theorist, I'm sure--is that he relies on statistics all over the place. He just doesn't know what he's talking about--and, even worse, doesn't seem to know that he doesn't know.

I mouth off all the time about things I don't know about. But at least when I go on about Karl Popper, for example, I ground it in my own experience as a researcher, I don't just spout off in general.

Anyway, my point is not to pick on Runciman for a year-old article that he probably whipped off in a couple of hours and maybe already regrets. I'm just using it as an example of how people who don't know statistics are doomed to rely on statistics all the same.

Just as Bill James pointed out how fans who hate sabermetrics (and all it stands for) were forming all sorts of misinformed opinions based on batting averages and the like.

Chris Bowers writes:

The nation still moving away from Republicans demographically, too. It can't be emphasized enough that Michael Dukakis would have won the 2008 election. His exit polls of 40% among whites, 89% among African-Americans, and 70% among Latinos is enough to reach 50%+1 now, even in the event that African-American turnout was only 12% of the vote instead of 13%.

From our analysis of the Current Population Survey post-election supplement, here are our estimates for voter turnout in 2008: 76.4% white, 11.9% black, 7.4% hispanic, 4.3% other, with the categories defined as mutually exclusive (for example, if you're white and hispanic, you count as "hispanic"). The exit polls say 74% white, 13% black, 9% hispanic, and 5% other (not adding to 100% because of rounding error), but I think CPS is more trustworthy.

Now we can take the Dukakis numbers and plug them into the 2008 turnout numbers, as long as we make some estimate for the votes of "other." I'll assume 55%, halfway between his performance among whites and among hispanics. (By comparison, we estimate from the Pew pre-election polls that Obama got 45% of the two-party vote among whites, 96% among blacks, 68% among hispanics, and 59% among others.)

Plugging in Dukakis's percentages by ethnic group and using the turnout numbers of 2008, we get a national adjusted Dukakis vote of .40*76.4% + .89*11.9% + .70*7.4% + .55*4.3% = 48.7%, which is better than the 46.1% he actually received but not quite enough to win.

This doesn't really shoot down Bowers's main argument--demographic shifts are important. I think he was overstating his case just slightly.

And, yes, I know that if Dukakis had really been running in 2008, things would've been different. I'm just following Bowers in using the Dukakis vote as a handy way to summarize the trends, keeping voting by ethnic group constant. Voting by ethnic group is not constant (as we can see by comparing Obama's breakdowns to his predecessors), but doing this sort of calculation is a good way to visualize the demographic changes that are occurring.

Daniel Becker's Random-Walk graphically demonstrates how different distributions can be generated with physical processes: Normal distribution falls out of a Pachinko machine, and Poisson from a dart-throwing process. He also shows how pseudo random number generators have higher-order correlations within them. Pretty!

[via Infosthetics]

Robert Frank defends carbon offsets at the sister blog. I'm sympathetic to much of Frank's argument; in particular, the fact that Al Gore has a big house isn't much of an argument against carbon offsets. (If the crops are failing and the flood waters are rising, it won't be much help to stand on a street corner shouting: But Al Gore had a big house!)

But I'm not happy with the example that Frank chooses to illustrate his point. He writes:

Mojca has pointed me to Paul Nylander's visualizations. He's using raytracing software and Mathematica to create pieces of visualization art:

I tried looking for examples that could be useful in the usual statistical practice, and his example of Horseshoe magnetic fields demonstrates distributions on a surface:

Henry reports that a colleague of his at George Washington University, Jeffrey Rosen, says he has sworn off blogging for good. But then Henry asks whether Rosen was really a blogger at all: in Henry's words:

A 1,000 word commissioned essay for The New Republic, which goes through its usual editorial processes, is usually not considered a 'blog entry.' . . . I [Henry] will say that I'd prefer not to see the term blogpost become a residual category for 'stuff I wrote which I wish I had thought through a bit more before I hit send.'

I just have two comments on this intra-GW conflict:

1. Does blogging now have higher prestige than magazine writing? It used to be that newspaper and magazine writers were insisting that blogging wasn't journalism. Now we have unpaid bloggers saying that magazine writing isn't really blogging!

2. Given the New Republic's history, I wouldn't say that "its usual editorial processes" counts for much. I'm guessing I have a more rigorous vetting process on my own blog (where the rule is that anyone with access can post any time) than the New Republic has for its thousand-word commissioned essays.