July 2006 Archives

I just reviewed a paper for a statistics journal. My review included the following sentences which maybe I should just put in my .signature file:

The main weakness of the paper is that it does not have any examples. This makes it hard to follow. As an applied statistician, I would like an example for two reasons: (a) I would like to know how to apply the method, and (b) it is much easier for me to evaluate the method if I can see it in an example. I would prefer an example that has relevance for the author of the paper (rather than a reanalysis of a "classic" dataset), but that is just my taste.

Lest you think I'm a grouch, let me add that I liked the paper and recommended acceptance. (Also, I know that I do not always follow my own rules, having analyzed the 8 schools example to death and having even on occasion reanalyzed data from Snedecor and Cochran's classic book.)

Peter Yared pointed me to these maps that he made showing various characteristics of U.S. states (from Census data) that have similar patterns to the votes for Democrats and Republicans in recent Presidential elections (that is, comparisons of the coasts and industrial midwest to the southern and central states). Many of these relate to the well-known recent correlation between state income and support for the Democrats. The patterns at the individual levels may differ, though. At a policy level, it makes sense that the Republicans favor transfer payments to poor states but not to poor people; with the reverse pattern for the Democrats.

This article by Donald Kinder is an interesting review of research on political views as they are inherited and as they develop with age. I also like it because he refers to my sister's work on the essentialist beliefs of children.

I'll have to think a bit about how this all relates to this picture of party ID and age:

(see also the other data here).

I received the following (unsolicited) email:

Eduardo linked to this interesting paper by Walter Mebane on using Benford's Law (the distribution of digits that arises from numbers that are sampled uniformly on a logarithmic scale) to investigate election fraud. I'll give my thoughts, but first here's the abstract:

This paper, "Biological versus nonbiological older brothers and men's sexual orientation," by Anthony Bogaert, appeared recently in the Proceedings of the National Academy of Sciences and was picked up by several news organizations, including Scientific American, New Scientist, Science News, and the CBC. As the Science News article put it,

The number of biological older brothers correlated with the likelihood of a man being homosexual, regardless of the amount of time spent with those siblings during childhood, Bogaert says. No other sibling characteristic, such as number of older sisters, displayed a link to male sexual orientation.

I was curious about this--why older brothers and not older sisters? The article referred back to this earlier paper by Blanchard and Bogaert from 1996, which had this graph:

and this table:

Here's the key quote from the paper:

Significant beta coefficients differ statistically from zero and, when positive, indicate a greater probability of homosexuality. Only the number of biological older brothers reared with the participant, and not any other sibling characteristic including the number of nonbiological brothers reared with the participant, was significantly related to sexual orientation.

The entire conclusions seem to be based on a comparison of significance with nonsignificance, even though the differences do not appear to be significant. (One can't quite be sure--it's a regression analysis and the different coef estimates are not independent, but based on the picture I strongly doubt the differences are significant.) In particular, the difference between the coefficients for brothers and sisters does not appear to be significant.

What can we say about this example?

As I have discussed elsewhere, the difference between "significant" and "not significant" is not itself statistically significant. But should I be such a hard-liner here? As Andrew Oswald pointed out, innovative research can have mistakes, but that doesn't mean it should be discarded. And given my Bayesian inclinations, I should be the last person to discard a finding (in this case, the difference between the average number of older brothers and the average number of older sisters) just because it's not statistically significant.

But . . . but . . . yes, the data are consistent with the hypothesis that only the number of older brothers matters. But the data are also consistent with the hypothesis that only the birth order (i.e., the total number of older siblings) matters. (At least, so I suspect from the graph and the table.) Given that the 95% confidence level is standard (and I'm pretty sure the paper wouldn't have been published without it), I think the rule should be applied consistently.

To put it another way, the news articles (and also bloggers; see here, here, and here) just take this finding at face value.

Let me try this one more time: Bogaert's conclusions might very well be correct. He did not make a big mistake (as was done, for example, in the article discussed here). But I think he should be a little less sure of his conclusions, since his data appear to be consistent with the simpler hypothesis that it's birth order, not #brothers, that's correlated with being gay. (The paper did refer to other studies replicating the findings, but when I tracked down the references I didn't actually see any more data on the brothers vs. sisters issue.)

Warning: I don't know what I'm talking about here!

This is a tricky issue because I know next to nothing about biology, so I'm speaking purely as a statistician here. Again, I'm not trying to slam Bogaert's study, I'm just critical of the unquestioning acceptance of the results, which I think derives from an error about comparing statistical significance.

David Kane writes,

I [Kane] am putting together a class on Rhetoric which will look the ways we use words, numbers and pictures to persuade. The class will have mininal prerequisites (maybe AP Stats or the equivalent) and will be discussion/tutorial based. For the sections on numbers and pictures, I plan on assigning "How To Lie with Statistics" by Huff and "The Visual Display of Quantitative Information" by Tufte. (The students will also be learning R so that they can produce some pretty pictures of their own. The course objectives are ambitious.)

Question: What other readings might people suggest? I am especially interested in readings that are either "classic" or freely available on the web.

I hope to teach the students are to attack and defend things like the IPCC report on global warming, the EPA report on secondhand smoke and so on.

Any suggestions would be much appreciated.

My response: I can't actually think of any great examples, partly because once an issue seems clear, one way or another, persuasive reasoning seems almost a separate issue from quantitative reasoning. (I suppose this is parallel to the idea in science that if you do an experiment well, you don't need statistics because the result will jump out at you.)

OK, I'll give one recommendation: chapter 10 of my book on teaching statistics. Here are the table of contents and index for the book. I really like the index--I think it's actually fun to read.

You ask for resources on the web. I suppose it won't hurt to post one chapter . . . so here's the aforementioned Chapter 10. (The images are clearer in the published book, but I think the pdf gives the general impression.) I hope you find it useful.

Frederick Mosteller passed away yesterday. He was a leader in applied statistics and statistical education and was a professor of statistics at Harvard for several decades. Here is a brief biography by Steve Fienberg, and here are my own memories of being Fred's T.A. in the last semester that he taught statistics. I did not know Fred well, but I consider him an inspiration in my work in applied statistics and statistical education.

A Bayesian prior characterizes our beliefs about different hypotheses (parameter values) before seeing the data. While some (subjectivists) attempt to elicit informative priors systematically - anticipating certain hypotheses and excluding others - others (objectivists) prefer noninformative priors with desirable properties, letting the data "decide". Yet, one can actually use the data to come up with an informative prior in an objective way.

Let us assume the problem of modeling the distribution of natural numbers. We do not know the range of natural numbers a priori. What would be a good prior? A uniform prior would be both improper (non-normalized), but also inappropriate: not all natural numbers are equally prevalent. A reasonable prior would be an aggregate of natural number distributions across a large number of datasets.

Dorogovtsev, Mendes and Oliveira have used Google for assessing the Frequency of occurrence of numbers in the World Wide Web. While they were not concerned about priors, their resulting distribution is actually a good general prior for natural numbers. Of course, it would help knowing if the natural numbers are years or something else, but other than that, the general (power law) distribution of p(n) ~ 1/sqrt(n) is both supported by data and mathematically elegant:

In this context it is worth mentioning also Benford's law, which elaborates on an observation that the leading digits are not equally likely. Instead, 1 is considerably more likely than 9.

I came across this picture posted by Steve Hulett on the Animation Guild blog:

(See here for a bigger image.) Uncle Woody (he was born in 1916 and was named after Woodrow Wilson. It could've been worse--his sister Lucy was named after Luther Burbank, and her full name is Lutheria) worked in animation--we were always told that he drew the "in-between" drawings for cartoons--and worked on promotions for Topps gum for many years:

Wacky Packs were the biggest thing when I was in elementary school but I never knew that Uncle Woody had worked on them.

Using multilevel modeling of state-level economic data and individual-level exit poll data from the 2000 Mexican presidential election, we find that income has a stronger effect in predicting the vote for the conservative party in poorer states than in richer states---a pattern that has also been found in recent U.S. elections. In addition (and unlike in the U.S.), richer states on average tend to support the conservative party at higher rates than poorer states. Our findings are consistent with the 2006 Mexican election, which showed a profound divide between rich and poor states. Income is an important predictor of the vote both at the individual and the state levels.

Here's the paper, and here's the key graph:

The little circles in the plots show the data from the exit poll from the 2000 election (average vote plotted vs. income category within each state, with size of the circles proportional to the number of survey respondents it represents). Party is coded as 1=PRD, 2=PRI, 3=PAN, so higher numbers are more conservative. The solid line in each plot represents the estimated relation between vote choice and income within the state (as fitted from a multilevel model). The gray lines represent uncertainty in the fitted regression lines.

The graph shows the 32 states (including Mexico, D.F.) in increasing order of per-capita GDP. The slopes are higher--that is, income is a stronger predictor of the vote--in poor states. Income is a less important predictor in the rich states (except for the capital, Mexico, D.F., which has its own pattern).

Here's a plot of the slopes vs. per-capita GDP in the 32 states:

The conservative party did better with rich voters everwhere, but individual income is a much stronger predictor of the vote in poor states than in rich states. This is similar to the pattern we found in the U.S. One difference between the two countries is that in the U.S., the conservative party does better in the poor states, but in Mexico, the conservative party does better in the rich states. But at the level of individual voting, the patterns in the two countries seems similar.

We plan to replicate our study with 2006 exit polls, once we can get our hands on the data.

I was just fitting a model and realizing that some of the graphs in my paper were all wrong--we seem to have garbled some of the coding of a variable in R. (It can happen, especially in multilevel models when group indexes get out of order.) But the basic conclusion didn't actually change. This flashed me back to when Gary and I were working on our seats-votes stuff (almost 20 years ago!), and we joked that our results were invariant to bugs in the code.

Sheena Iyengar is a professor of psychology in the business school here who has worked on some interesting projects (including the speed-dating experiment). She writes,

I [Sheena] am looking for an ambitious, dedicated, and promising graduating senior interested in a full-time research assistant position for one to two years beginning August 1, 2006. Potential applicants should have a degree in either social/cognitive psychology or economics with an interest in the intersection of economics and the psychology of judgment and decision making. Preference is given to candidates with a strong math background and good writing skills who have had some research experience in a laboratory already.

The salary for this position is $45,000 and includes all health benefits. The research assistant will be responsible for running experiments, managing a laboratory, conducting statistical analyses, and will have the opportunity to co-author in journal publications. It is a truly excellent opportunity for someone who is interested in pursuing a Ph.D. in behavioral economics, psychology, and/or related disciplines. If you are interested in applying for this position, please e-mail me, Professor Sheena S. Iyengar, at ss957@columbia.edu or call at 212 854-8308. I will be interviewing potential applicants immediately.

It looks interesting to me . . .

In googling for "parking lot Stolzenberg," I came across a series of articles in the American Sociological Review on the measurement of church attendance in the United States--an interesting topic in its own right and also a great example for teaching the concept of total survey error in a sampling class. The exchange begins with an article by C. Kirk Haraway, Penny Long Marler, and Mark Chaves in 1993:

Characterizations of religious life in the United States typically reference poll data on church attendance. Consistently high levels of participation reported in these data sug-gest an exceptionally religious population, little affected by secularizing trends. This picture of vitality, however, contradicts other empirical evidence indicating declining strength among many religious institutions. Using a variety of data sources and data collection procedures, we estimate that church attendance rates for Protestants and Catholics are, in fact, approximately one-half the generally accepted levels.

Jorge Lopez sent me the following report analyzing results from the recent Mexican election. He looked at the vote totals as they emerged through the election night, and saw patterns that led him to conclude that there was fraud in the vote counting. His report begins:

Many of us took advantage of the latest technology and followed last Sunday’s elections in Mexico through a novel method: web postings of the votes through the Program of Preliminary Results, or PREP by its Spanish initials. What Mexico’s Federal Electoral Institute (IFE) did not take into account is that the postings were not only informing, they were providing valuable data that can be –and was- examined to check its “health”. The bottom line is that the data presented is ill, so ill that it appears to have been given artificial life by a computer algorithm.

What the web surfers saw is that after an initial strong showing, which began at Sunday noon with a Calderon advantage of more than 4% over López Obrador (“AMLO”), the lead began to decrease in percentages. The diminishing trend continued and, around midnight, many of us went to bed forecasting a tie by 3:00 AM Monday, and an AMLO advantage of about 1% by wake up time on Monday. The morning surprise was that the trend had changed overnight and Calderon appeared with a slim but invariant advantage of about 1%; this sent many of us to what we, physics professors, do for a living: data analysis.

. . .

Here's the full report.

I looked at the report, and I don't think it represents convincing evidence of data manipulation. There are three reasons why I say this:

1. The report doesn't have information on where the election returns came from. Thus, the changes in the votes (going one way, then reversing, etc, as shown on page 4) could arise from votes coming from different places.

2. It's not such a surprise that the vote total will become more stable over time, because the vote total at time t+1 mostly comes from the vote total at time t. So I don't see the correlation of .9999 as necessarily being meaningful.

3. In the picture on page 3, there's no particular reason to expect a normal distribution. You will see differences of close to zero in percent as the counts go on over time.

The data being analyzed remind me of an analysis I did a few years ago of a local election in New York City; see this paper ,which appeared in Chance (and also will appear in Chapter 2 in our forthcoming book).

As I told Jorge, although I disagree with his conclusions, it's good to air these things and let people make their own judgments, hence this blog posting. Jorge also send me this document from Eduardo Trejo which has some of the preliminary vote counts. Jorge also asked that if anyone has any comments, they can post them on this blog and also can send him email.

(For some more background on allegations of fraud in the Mexican election, see this Boingboing entry by Xeni Jardin, which has link to more stuff.)

I'm sometimes asked if I can recommend a statistical consultant. Rahul Dodhia is a former student (and coauthor of this paper on statistical communication) who, after getting his Ph.D. in psychology, has worked at different places including NASA, Ebay, and Amazon. He does statistical consulting; see here. I also have some colleagues in the Columbia faculty who do consulting. Rahul's the one with the website, though.

I have always been taught that the randomized experiment is the gold standard for causal inference, and I always thought this was a universal view. Not among all econometricians, apparently. In a recent paper in Sociological Methodology, James Heckman refers to "the myth that causality can only be determined by randomization,
and that glorifies randomization as the ‘‘gold standard’’ of causal inference."

It's an interesting article because he takes the opposite position from all the statisticians I've ever spoken with (Bayesian or non-Bayesian). Heckman is not particularly interested in randomized experiments and does not see them as any sort of baseline, but he very much likes structural models, which statisticians are typically wary of because of their strong and (from a statistical perspective) nearly untestable assumptions. I'm sure that some of this dispute reflects different questions that are being asked in different fields.

Heckman's article is a response to this article [link fixed--thanks Alex] by Michael Sobel, who argues that Heckman's methods are actually not so different from the methods commonly used in statistics. It's all a bit baffling to me because I actually thought that economists were big fans of randomized experiments nowadays.

P.S. As noted by an anonymous commenter, some controversy arose from this issue of Sociological Methodology, but I'm not going into detail here since said controversy is not very relevant to the scientific issues that arise in these papers, which is what I wanted to post on.

Christian Robert sent this along:

Post-doctoral position in statistical cosmology

ECOSSTAT Program National Research Agency - CNRS: Measuring cosmological parameters from large heterogeneous surveys

The ECOSSTAT program is an inter-disciplinary three year project between astrophysicists and statisticians that aims at refining the constraints on the values of parameters in the cosmological model.

Cajo Ter Braak just published this paper in Statistics and Computing. It's an automatic Metropolis-like algorithm that seems to automatically work to perform adaptive jumps. Perhaps could be useful in a Umacs or Bugs-like setting? Here's the abstract:

OK, this one is for hard-core Bayesians only . . . it's some info from Brad Carlin, Nicky Best, and Angelika van der Linde on the deviance information criterion (DIC):

Graham Webster pointed me to this interesting site that's full of data and graphs. Should be great for teaching, and for research too, in enabling people to look up and graph data quickly.

I'd like to develop some homework assignments and class-participation activities based on this site. We should be able to do better than to tell students: Hey, look at this, it's cool!

To start with, one could set students on to it and ask them to find pairs of variables with negative correlations, or pairs of variables that are approximately independent, or pairs that have zero correlation but are not independent. Or one student could pick a pair of variables, and the other could guess the regression slope.

I'm sure more could be done: the challenge is to get the students to be thinking hard, and anticipating the patterns before they see the data, rather than simply passively looking at cool patterns.

Andrew Oswald (author of the paper that found that parents of daughters are further to the left, politically, than parents of sons) writes,

I read your post on Kanazawa. I don't know whether his paper is correct, but I wanted to say something slightly different. Here is my concern.

The whole spirit of your blog would have led, in my view, to a rejection of the early papers arguing that smoking causes cancer (because, your eloquent blog might have written around 1953 or whenever it was exactly, smoking is endogenous). That worries me. It would have led to many extra people dying.

I can tell that you are a highly experienced researcher and intellectually brilliant chap but the slightly negative tone of your blog has a danger -- if I may have the temerity to say so. Your younger readers are constantly getting the subtle message: A POTENTIAL METHODOLOGICAL FLAW IN A PAPER MEANS ITS CONCLUSIONS ARE WRONG. Such a sentence is, as I am sure you would say, quite wrong. And one could then talk about type one and two errors, and I am sure you do in class.

Your blog is great. But I often think this.

I appreciate it is a fine distinction.

In economics, rightly or wrongly, referees are obsessed with thinking of some potential flaw in a paper. I teach my students that those obsessive referees would have, years ago, condemned many hundreds of thousands of smokers to death.

I replied as follows:

Jorge Bravo pointed me to this report by some statisticians (Miguel Cervera Flores, Guillermina Eslava Gomez, Ruben Hernandez Cid, Ignacio Mendez Ramirez, and Manuel Mendoza Ramirez) on the very close Mexican election. (Here's the report at its original url.)

Here are their estimated percentages for each party:

and here's the graphical version, just comparing PAN to PRD:

I can't actually figure out exactly where these estimates come from, or what exactly they are doing to get the robust, classical, and Bayesian estimates. But they should give their estimates for the difference between the two leading parties, I think, rather than separate intervals for each.

I just heard, on the radio, the recent Mexican election, which was almost tied, described as a "worst case scenario." It's a funny thing, though: a very close election is a bad thing in that it can lead to controversy, lack of legitimacy of the government, and sensitivity of results to cheating. On the other hand, a key premise of democracy is that your vote can matter, which means there has to a be a chance that the election is really close. So, the ideal seems to be an election that, before the election, is highly likely to be close, but after the election, never ends up actually being close. This is hard to arrange, though!

It's a paradox, along the lines of: you should live each day as if it were your last, but you don't want it to actually be your last...