Responded:

John Clute
Richard Florida
Malcolm Gladwell
Sander Greenland
Daniel Gross
Mickey Kaus
Paul Krugman
Andrew Leonard
John Lott
Jay Nordlinger
Andrew Oswald
Ed Park
Steve Sailer
John Seabrook
Nassim Taleb
Josh Tenenbaum

Did not respond:

Robert Frank
Satoshi Kanazawa
George Packer
Russ Alan Price
David Runciman

I think I've missed a few here (in both categories). Also, some people I'm still waiting to hear from, and some respond but not in a useful way.

P.S. I just noticed: all these people are male (and most are white)! I'll have to diversify a bit!

Colorado, Connecticut and New Jersey are at the top. Colorado was a battleground state.

I had a question for you about the youth vote. What are its ethnic and red/blue composition? The reason I ask is that I was trying to integrate the apparently growing Democratic dominance in this segment with various other beliefs I have seen expressed, e.g

a) that red states have larger fertility (affordable family formation or whatever)

b) that families have an impact on the political beliefs of children (more than educators, as educators insist - at least at the college level, I haven't really seen a discussion of school teachers) which would then provide a mechanism for (a) to affect voting share to the right of the spectrum

c) that the minorities form a growing share of the young which would tilt the playing field to the left.

My reply:

1. I don't yet have raw survey data. The exit polls on the web do break down the vote by age and race. Among blacks, Obama won about the same among all age groups. Among Hispanics, Obama did 8% better among the young than the old, and among whites, Obama did 14% better among the young than the old.

But . . . if you believe the exit polls (which I don't, completely), there was an interaction between age and race: many more of the young voters were ethnic minorities. Among blacks and Hispanics, there were three times as many under-30's as over-65's. (By comparison, among whites, there were more old voters than young voters.)

So the age effect partly arose from lots of young ethnic minorities coming out to vote.

2. People do tend to vote like their parents--children of Republicans are, on average, more likely to vote Republican--but cohort effects go on top of this. The recent economy and George W. Bush's approval ratings aren't likely to make the Republican Party popular with young people--especially those who are ethnic minorities. Any differences in birth rates between states are small compared to these big political swings, which are not just about Obama; see this graph from 2006:

Why did [his] strategy work . . . it's hard to escape the conclusion that . . . there are times when being an outsider is precisely what makes you a good insider.

Later, he continues:

It’s one thing to argue that being an outsider can be strategically useful. But Andrew Carnegie went farther. He believed that poverty provided a better preparation for success than wealth did; that, at root, compensating for disadvantage was more useful, developmentally, than capitalizing on advantage.

At some level, there's got to be some truth to this: you learn things from the school of hard knocks that you'll never learn in the Ivy League, and so forth. But . . . there are so many more poor people than rich people out there. Isn't this just a story about a denominator? Here's my hypothesis:

Pr (success | privileged background) >> Pr (success | humble background)

# people with privileged background << # of people with humble background

Here's more from Gladwell's article:

Continued reading The Denominator, or, Is it an advantage to have a humble background?...

Posted by Andrew at November 21, 2008 2:32 PM

TrackBack (0) | Comments (12)

But his progress had slowed to a crawl. [...] Bertoni says it’s partly because of “Napoleon Dynamite,” an indie comedy from 2004 that achieved cult status and went on to become extremely popular on Netflix. It is, Bertoni and others have discovered, maddeningly hard to determine how much people will like it. When Bertoni runs his algorithms on regular hits like “Lethal Weapon” or “Miss Congeniality” and tries to predict how any given Netflix user will rate them, he’s usually within eight-tenths of a star. But with films like “Napoleon Dynamite,” he’s off by an average of 1.2 stars.

The reason, Bertoni says, is that “Napoleon Dynamite” is very weird and very polarizing. [...] It’s the type of quirky entertainment that tends to be either loved or despised.

And here is the stunning conclusion by fortunately anonymous computer scientists:

Some computer scientists think the “Napoleon Dynamite” problem exposes a serious weakness of computers. They cannot anticipate the eccentric ways that real people actually decide to take a chance on a movie.

Actually, computers do quite a good job modeling probability distributions for those more eccentric and unpredictable of us. Yes, the humble probability distribution, the centuries-old staple of statisticians is enough to model eccentricity! The problem is that Netflix makes it hard to use sophisticated models the scoring function is the antiquated and not just pre-Bayesian but actually pre-probabilistic root mean squared error or RMSE. For all practical purposes, the square root in RMSE is a monotonic transformation that won't affect the ranking of recommender models, and we can drop it outright.

So, if one looked at the distribution of ratings for Napoleon Dynamite on Amazon, it has high variance:

On the other hand, Lethal Weapon 4 ratings have lower variance:

If we use the average number of stars as the context-ignorant unpersonalized predictor (which I've discussed before), ND will give you mean squared pain of 3.8, and LW4 will give you the mean squared pain of 2.7. Now, your model might choose not to make recommendations with controversial movies - but this won't help you on Netflix Prize - you're forced to make errors even when you know you're making them. (R)MSE is pre-probabilistic: it gives no advantage to a probabilistic model that's aware of its own uncertainty.

Continued reading Still another 10 days to apply for an Earth Institute postdoc...

Posted by Andrew at November 20, 2008 9:42 PM

TrackBack (0) | Comments (0)

Could you tell me what is the difference between "uncertainty" and "ignorance" in this theory [of belief functions]? Some authors define "ignorance" as the "uncommitted belief" which is assigned to the whole frame of discernment, others define it as the difference between Plausibility and Belief (Pl() - Bel()). Some authors define value assigned by Belief function for elements as "uncertainty".

I don't know. All I know about belief functions is in my article about the boxer, the wrestler, and the coin flip, which is actually a writeup of something I did 20 years ago. So no new thoughts, unfortunately.

Frequentist Probability Probability is easily justified as a very elegant way of dealing with uncertainty in cases and variables. But probability is not observed directly but instead inferred - as are the parameters in contrast to observable predictors and outcomes. Frequentists state that the probability should be measured through the gold standard of an infinite sequence of observations, and question the benefit of Bayesian approach while criticizing the fact that inferring a parameter Bayesianly can yield worse accuracy than their favored method of "estimators" - and a bad prior can totally mess up inference. So why not use estimators if their asymptotic properties are good and the methodology often simpler than Bayes?

Overfitting Dividing the number of positive outcomes with the number of all outcomes to estimate the probability of the positive outcome is a very simple estimator: it's easy to have enough data to calculate this. But most interesting questions are not as simple: it is not interesting to calculate the probability of getting cancer, and the probability of getting cancer given smoking also requires removing the obvious effect of age. All these additional variables make a model more complicated, and the number of parameters greater. Without care and attention the model can start hallucinating properties that aren't there. The problem is shown in the following picture:

If your modeling problem is in the green area, you can happily use estimators or maximum likelihood. If you're entering the yellow area and want to retain some generalization power, you need some sort of regularization, epitomized by L1 and L2 regularization, AIC, feature selection or support vector machines. So why shouldn't we just regularize?

Priors Priors are how a Bayesian would perform regularization. After seeing a large number of regression problems from medical domains, we can safely assign a prior distribution to the size of a regression coefficient, as we have done in our paper. But then, what is the advantage over regularization? A prior is just a distribution of what the parameters should be over a particular category of problems! Isn't this a nice way to formulate regularization?

Model Uncertainty The crux of Bayes is in using probability to represent the uncertainty about the Platonic - the model, its parameters, the probability. The Bayesian approach truly starts paying a dividend when there is uncertainty in models and parameters, when we have insufficient data to accurately fit the model. Even if an estimator could rather accurately match the predictions obtained by a posterior, the variance in the posterior allows us to understand when the model can't be fit. To the best of my knowledge, no other methodology can automatically detect such problems.

Another problem that Andrew identified is that there might be situations where the data doesn't match the model very well - and even though there might be lots of data and a relatively simple model - it just doesn't fit, and the posterior will be vague.

Language of Modeling WinBUGS is an example of a higher-level modeling language. Just as programming languages have been celebrated as improving programmers' productivity: they do not require the programmer to think in terms of individual statements such as SET or JMP but in terms of functions, procedures, loops. Similarly, with Bayesian models we no longer have to think in terms of derivatives and fitting algorithms, but in terms of parameters having distributions and tied together in models. Gibbs sampler is a general-purpose fitter and proto-compiler. Of course, it's not nearly as efficient as a hand-written optimizer, but in the future tools like the Hierarchical Bayes Compiler (HBC) will create custom fitters given a higher-level specification of the model.

Summary The primary value of the Bayesian paradigm is its formal elegance which allows automation of key problems: probability takes care of unpredictability in phenomena, priors help prevent overfitting by providing outside experience (AI practitioners would refer to it as background knowledge), the use of model uncertainty helps determine the reliability of predictions, and applied Bayesians are beginning to develop model compilers!

Future The theory and practice of data analysis is currently all mixed up among a number of overlapping disciplines: (applied/mathematical/geo/medical/...)statistics, machine learning, data mining, (econo/psycho/bio)metrics, bioinformatics. All of them pursue the same problems with different but qualitatively similar tools, lacking the scale to build tools that would help them get to the next level. It is important to disentangle them. The future of data analysis should lie on these four fronts:

1. reliable compilers and samplers that will work with large databases, provide reliable sampling (see BUGS, HBC - empowered by the new generation of programming languages such as Haskell)
2. internet databases intended to manage background knowledge and related data sets, where the same variable appears and the same phenomenon appear in multiple tables, allowing priors to be based on more than a single data set. Research should be presented as raw data in a standardized form, not as reports and aggregates that prevent others from building on top of the finished work. Too many people are working on the same problems but not sharing the data because of an unsolved issue of the rights of the collectors of data who can only gain credit for publications (see FreeBase, Machine Learning Repository, Trendrr, Swivel, OECD.Stat)
3. visualization & modeling environments that make it easier to clean and transform data, experiment with models, to present insights, to reduce the amount of time needed to turn data into a model that can be communicated. (see R Project, Processing, Gapminder)
4. interpretable modeling is important to bring formal models closer to human intuition. It is still not clear what is the importance of a predictor for the outcome - the regression coefficient is close, but yet often confusing. With more powerful modeling frameworks, it is going to be possible to focus on this - not being worried about what one can fit, but instead with model choice, model selection, model language, visual language.

What do you think? What links did we miss?

There is also little consensus among researchers about what causes psychopathy. Considerable evidence, including several large-scale studies of twins, points toward a genetic component. Yet psychopaths are more likely to come from neglectful families than from loving, nurturing ones.

I'm confused here. If there's a big genetic component, wouldn't it stand to reason that parents of psychopaths are more likely to be neglectful and less likely to be loving and nurturing? So why the "Yet" in the quote above? Or is there something I'm missing?

P.S. in response to commenters: Yes, I agree that it's possible for psychopathy to be largely genetic without parents of psychopaths being much more likely to be neglectful.

What I didn't understand was Seabrook's implication that this would be surprising, the idea that if (a) a trait is genetically linked, and (b) a trait can be (somewhat) predicted by parental behavior, that the combination of (a) and (b) should be considered puzzling. By default, I'd think (a) and (b) would go together.

Depressive symptoms are significantly higher amongst left-handed men. While 19% of right handed men report experiencing depressive symptoms for at least a two week period, the figure for left handed men is almost 25%. For women the corresponding percentages are 33% and 36% respectively but the difference is not statistically significant.

The analysis is of "a new large population survey from twelve European countries," a random sample of 27000 non-institutionalized people aged 50 and older. Handedness was classified based on self-reporting, and depression is measured using standard questions. Of the sample, about 7% of men and 6% of women were classified as left-handed.

My only suggestion (beyond reporting fewer significant digits in the tables) is to rescale the depression scale by dividing by two standard deviations; this would allow the coefficients to be interpretable on the same scale as those for the binary outcome (see Table 2).

I [Ben] had this map [see below] on my door for the last week. Based on exactly the same calculation using constant 95% black support and census-proportional representation. The white counties are the ones whose census names didn't match properly with the names used in the library(maps) package in R, I was too lazy to fix them.

Cool. I'd only suggest using light gray rather than heavy black lines between counties; the map as it is overemphasizes the county borders, I think. But I respect his laziness; there's always time later to fix the details.

Ben continues:

[Below are] the state-by-state county share plots for the lower 49, Obama vote share as a function of black population share. V.O. Key's observation that whites who live near blacks in southern states are less positively inclined towards them is *still* visible in several states.

The circle areas are proportional to county voter turnout. (The biggest circle is L.A. county in California, and so forth.)

Ben also had this comment about his map:

It reminded me of something Bob Putnam would say every time someone presented an empirical talk in our Center for the Study of Democratic Politics series during the year he was a fellow here at Princeton: "You should include miles to the Canadian border as a variable in your regression, it is the most important proxy for political culture in America!" At least in the eastern half of the country, he has a point.

Except for New Hampshire and Vermont, I think.

P.S. For graphics enthusiasts, here are some earlier graphs that I gave the thumbs-down on before Ben came up with the 50 plots above:

Continued reading Estimated votes by county among non-blacks...

Posted by Andrew at November 18, 2008 4:25 PM

TrackBack (0) | Comments (4)

My latest thought is Little Children by Tom Perotta. It's an excellent book but it's not a great work of art, but that's the point: when teaching a class, maybe it's better to have something where the seams show a little.

P.S. See comments below. Also, Bridget Jones's Diary. And some kids' humor book: not something like Lemony Snicket that's supposed to be good, but something more lowbrow such as Goosebumps or Captain Underpants, to get a sense of what people think is funny. Also, something funny but completely non-novel-like, for example Chris Rock's book. Students can compare how the comic novels differ from the quick jokes.

The article entitled, "Bayesian Analysis for Intelligence: Some Focus on the Middle East," was written by Nicholas Schweitzer . . . JIOX provides no information on the essay's origins, but . . . it appears to be a declassified CIA piece written sometime in the 1970's (note mentions of Presidents Asad and Sadat, and Prime Minister Rabin on page one). . . . Schweitzer concludes that in general the Bayesian technique was able to more quickly predict "non-events" (i.e., when no hostilities would occur among Middle Eastern nations) than analysts using only their expertise and intuitions. The research design included no baseline for comparison to an actual event; therefore, we are left wondering if the Bayesian technique described here would be able to predict when something will actually happen. Despite this obvious shortcoming, it is very encouraging to observe the level of sophistication being implemented by CIA analysts some thirty-odd years ago.

I actually participated a couple years ago in an (unclassified) meeting on Bayesian analysis for military intelligence, so I know that these ideas are still out there. My only comment, regarding the Bayesian issue per se, is that the key to good statistical methods is typically making use of relevant information; non-Bayesian methods can also be effective if they can be adapted to use the info that goes into a Bayesian procedure.

It's too bad, because my Red State, Blue State presentation is fun and informative, and I think the radio interviews I've done (with lengths ranging from 5 minutes to an hour) have gone well also. The two things that threw me off:
1. I've met Will before and I felt comfortable with him, hence too relaxed. Will was an excellent interviewer and gave me many opportunities to explain things; it wasn't his fault that I spouted off too much.
2. I've already spoken with Will about the book and so it was hard for me to remember to start from scratch--the audience won't necessarily be familiar with it.
3. Seeing my image in front of me while I was talking made me extra-focused on not twitching--always a bad thing. In a face-to-face or telephone interview, I usually forget about the twitching after a minute or so. Trying to suppress it takes a lot of mental effort that would be better used to think about my responses.

It would've been better to have some written talking points in front of me to keep me focused. The funny thing is, I did that for my early radio interviews but as I got more used to the format, I started speaking more off the cuff and it was going fine. This was just an interview too far. I had fun while it was happening, but afterward I realized what had gone wrong.

Anyway, it felt good to get this off my chest.