Results matching “R”

Jouni reports:

Dear all,

I'm sending this because you've been interested in my R packages 'rv' (simulation-based random variable class) and/or 'Umacs' (Universal Markov chain sampler).

The latest version of rv and the first "public beta" of Umacs are now available at CRAN (cran.r-project.org)

Please feel free to try them out!

http://cran.r-project.org/src/contrib/Descriptions/rv.html
http://cran.r-project.org/src/contrib/Descriptions/Umacs.html

The "rv" package is based on Jouni's thesis on fully Bayesian computing. The idea is to be able to manipulate random variable objects without having to get lost in the subscripting of simulations. (See here.)

Umacs is an automated way to program Bayesian computations (Metropolis algorithm and Gibbs sampler). It is roughly based on the function-based approach of the R programs in Appendix C of Bayesian Data Analysis.

Widgets are little rectangles people can easily "embed" into their web pages, social networking profiles and similar. A particularly interesting one was released a few days ago by the news aggregator Newsvine:

ElectionVine is a tiny polling widget that allows the visitors of a website to vote on different US presidential candidates. This wouldn't be anything new by itself, but what ElectionVine has done is aggregation from all these separate subpopulations into a joint display of political preferences, but also lists the results from individual virtual "polling stations".

For example, Facebook is not very politically biased, but many other websites are. These polls tell us a lot about the audience of particular web pages (and in reverse about the demographics), they probably don't tell much about the overall standing. In general, web population hasn't historically proven representative of that of the whole country. For example, Howard Dean was enormously popular on the Web, yet never made it beyond the primaries. I wonder if it is going to be the same case with Paul and Obama, who win at the web polls yet trail in the conventional surveys (for Rep or Dem). Perhaps it is because of the demographic slant in internet participation, that this BusinessWeek chart nicely displays.

There is some academic work on web-based polling, perhaps worth mentioning is a repository of web-based survey methods papers, WebSM.

As an aside, a few months ago McCain's MySpace page was using Mike Davison's (Newsvine co-founder) template without permission. As a prank, Davison then changed the template into a fake message.

Bill Harris writes,

Marc Tanguay writes,

I am senior economist at Statistics Canada and I have been involved since 6 years in building information systems for what we call here the System of National Accounts. This consists of a exhaustive collection of economics tables in many dimensions. The statistics concern employment, wages, hours worked, Gross domestics product (GDP), sales, imports, inventories, industrial uses, saving, household expenditures, etc. There is many dimensions in those tables as industries, sector, or geographical identities. The building of those tables involve a huge amount of information that are provided by dozens of sources of information; surveys, information from census and administrative data are the main providers.

A year or so ago I heard about a couple of papers by Satoshi Kanazawa on "Engineers have more sons, nurses have more daughters" and "Beautiful parents have more daughters."  The titles surprised me, because in my acquaintance with such data, I'd seen very little evidence of sex ratios at birth varying much at all, certainly not by 26% as was claimed in one of these papers.  I looked into it and indeed it turned out that the findings could be explained as statistical artifacts--the key errors were, in one of the studies, controlling for intermediate outcomes and, in the other study, reporting only one of multiple potential hypothesis tests.  At the time, I felt that a key weakness of the research was that it did not include collaboration with statisticians, experimental psychologists, or others who are aware of these issues.

We had this entry almost a year ago. This year Hans Rosling gives yet another talk titled "New insights on poverty and life around the world". The talk is great, and the ending is quite shocking..

In a follow-up to his now-legendary TED2006 presentation, Hans Rosling demonstrates how developing countries are pulling themselves out of poverty. He shows us the next generation of his Trendalyzer software -- which analyzes and displays data in amazingly accessible ways, allowing people to see patterns previously hidden behind mountains of stats. (Ten days later, he announced a deal with Google to acquire the software.) He also demos Dollar Street, a program that lets you peer in the windows of typical families worldwide living at different income levels. Be sure to watch straight through to the (literally) jaw-dropping finale.

Bernard Guerrero asks what I think of this. My response is that people can be irrational all the time--let's face it, we're a bunch of animals. Voters can have incoherent preferences (e.g., more services but less taxes), consumers can make mistakes (buying that brand-name $40,000 car and then being upset that they have no money left), forecasters can make mistakes (even setting aside "moral hazard" settings, there are lots of notorious problems such as people attaching insufficient probability to the "all else" category).

Arima models etc. can be overrated--lots of people seem to think these are the only models out there. Cavan Reilly has a fun example--chapter 27 in my book with Meng--of a 6-parameter predator-prey model that way outperforms standard time series models (with 11 or more parameters) in forecasting the famous Canadian lynx series. So I'm not surprised that Arimas can be beaten.

I agree with Bernard that you'd want to know where the survey forecasts come from. The surveys themselves are of forecasts. (This is different than the familiar use of surveys of forthcoming elections, where people are asked whom they would vote for if the election were held today. The Ang et al. paper is using surveys where people are explicitly asked to forecast.) It does sound like a classic "wisdom of crowds" averaging.

P.S. Two of the authors are at Columbia. I haven't met them. Perhaps they can speak in our quantitative social science seminar in the fall.

Michael Papenfus writes,

I have been using your ARM book to learn multilevel modeling and I have a question which I cannot seem to find answer to in your book. How can I assess the overall significance of a random effect (varying intercept)? Let say I estimate

M1 <- lmer(y ~ 1 + x (1|country)) and obtain
the results from your book which include

Error terrms:
Group Name Std.Dev.
county (Intercept) 0.33
Residual 0.76

Is there a way to test for the overall significance of the random effect (varying intercept)? I don't believe that the se.ranef is what I am looking for. I believe that some packages such as Stata xtmixed provide standard errors to the variance components.

My reply: the mcsamp() function does approximate posterior simulations for the multilevel model and will give you a sense of the uncertainty in the variance parameters. To test the statistical significance, though, you have to fit the model with and without the variance parameters and check some measure of fit. I haven't thought too much about this problem because I usually keep in whatever I can, discarding variance parameters only for convenience (in which case it's usually clear in practical terms that a variance component is small enough that it can be ignored). Ultimately, I think all the variance components exist at some level, although they can be small enough that they can be ignored. That said, you can compare models with and without variance components using predictive accuracy. See the table on the top of p.526 for an example.

I got this in the mail the other day. It's at the Institute for Tourism and Leisure Studies. Pretty cool (perhaps)! Certainly something unusual. I'm sure Bayesian data analysis, multilevel modeling, and statistical graphics will be useful in this job . . .

Rebecca sent in this example of a common statistical error:

Public opinion about the project seems guardedly supportive, with a majority of residents saying they favor it, though more than a quarter want its size to be reduced. The polls, taken for a local newspaper, use small samples, 500 people, limiting their usefulness as a gauge of popular sentiment in a city of one million.

Actually, if it's a random sample, then it's not a problem that the sample size is only a small fraction of the population size.

This by Freeman Dyson was pretty cool. Not the stuff about how open-source biotechnology is going to change the world--maybe he's right, maybe he's wrong, but it comes across to me as generic science writing. The cool stuff was his discussion of the ideas of Carl Woese (whom I'd never previously heard of):

[Woese asks] When did Darwinian evolution begin? By Darwinian evolution he means evolution as Darwin understood it, based on the competition for survival of noninterbreeding species. He presents evidence that Darwinian evolution does not go back to the beginning of life. When we compare genomes of ancient lineages of living creatures, we find evidence of numerous transfers of genetic information from one lineage to another. In early times, horizontal gene transfer, the sharing of genes between unrelated species, was prevalent. It becomes more prevalent the further back you go in time. . . .

In his "New Biology" article, he is postulating a golden age of pre-Darwinian life, when horizontal gene transfer was universal and separate species did not yet exist. Life was then a community of cells of various kinds, sharing their genetic information so that clever chemical tricks and catalytic processes invented by one creature could be inherited by all of them. Evolution was a communal affair, the whole community advancing in metabolic and reproductive efficiency as the genes of the most efficient cells were shared. Evolution could be rapid, as new chemical devices could be evolved simultaneously by cells of different kinds working in parallel and then reassembled in a single cell by horizontal gene transfer.

But then, one evil day, a cell resembling a primitive bacterium happened to find itself one jump ahead of its neighbors in efficiency. That cell, anticipating Bill Gates by three billion years, separated itself from the community and refused to share. Its offspring became the first species of bacteria—and the first species of any kind—reserving their intellectual property for their own private use. With their superior efficiency, the bacteria continued to prosper and to evolve separately, while the rest of the community continued its communal life. Some millions of years later . . . nothing was left of the community and all life was divided into species. The Darwinian interlude had begun.

Now this is cool--the idea that speciation itself is a sort of prisoner's dilemma, or killer app, so that once a species is formed, it can preserve its genetic identity and eventually outlast the faster-evolving but less walled-off organisms around them. Speciation has always been a mystifying aspect of evolution to me, so it's interesting to see this (possibly false, but interesting) theory.

Dan Goldstein writes, "Marketing is JDM [judgment and decision making] with teeth." Wouldn't that be dentistry?

John Kastellec points us to this book by Michael Crawley. It reminds me of what my old college professor wrote to me 12 years ago when I sent her a copy of my (then-)new book: "Thanks for sending this. A lot of people are writing books nowadays." (I have no comments on Crawley's book, having seen nothing but the link to the webpage.)

P.S. Somebody just said to me today, "Do you ever use R?" I said yes, and he said that he thought that applied people such as himself used R, but that the real statisticians used things like SAS. Grrrr.... I told him that SAS is said to have advantages with large datasets but isn't so great at model-fitting and graphics.

Song Qian sent me this paper, to appear in the journal Ecology, on ecological applications of multilevel analysis of variance. Here's the abstract:

A Bayesian representation of the analysis of variance by Gelman (2005) is introduced with ecological examples. These examples demonstrate typical situations we encounter in ecological studies. Compared to the conventional methods, the multilevel approach is more flexible in model formulation, easier to setup, and easier to present. Because the emphasis is on estimation, multilevel model results are more informative than the results from a significance test. The improved capacity is largely due to the changed computation methods. In our examples, we show that (1) the multilevel model is able to discern a treatment effect that is too weak for the conventional approach, (2) the graphical presentation associated with the multilevel method is more informative, and (3) the multilevel model can incorporate all sources of uncertainty to accurately describe the true relationship between the outcome and potential predictors.

I like the method (of course) and also the graphical displays. The next step is to move beyond exchangeable models, especially for interactions.

Kaiser discusses here the value of sketching preliminary versions of a plot to see what might work, before going to the effort of making the full graph. I agree completely--in my class on graphics, we would go through several mock-ups before trying to program something up.

The only trouble is that I don't know of any software for mockups. Ideally one could draw these prototypes and then feed in the data and see what the plots look like. A menu of 50 or so prototypes might do it, I suppose.

Manuel Spínola writes,

In the book you say that it is a good idea to plot the fitted model. How do you do that when you have, for example, 3 explanatory variables? Or do you mean plotting one variable at the time?

My reply:

I would plot one variable at a time, but then you can use multiple graphs to show different levels of a second variable, and multiple lines per graph to show different levels of a third variable, and solid/dotted lines to show a fourth variable, and a second dimension of a grid of plots to show a fifth, and color to show a sixth. For an example, see Figure 2 of this paper.

I used to like to use different symbols and symbol sizes to add more dimensions, but now I'm happier with "small multiples"--that is, one-way or two-way grids of plots. Sort of like Trellis, except that I like to label the internal axes locally (right on the little graphs) and label the external axes on the outside of the plot (see, for example, Figure 15.7 on page 335 in our book). I find the Trellis convention (using external labels for the internal axes) confusing.

More thoughts on the backseat driver principle from Ubs:

Anders Sandberg writes here about how the finish-the-plate bias can lead people to overeat, simply because food comes in larger packages (which, in turn, presumably arises because food is so cheap to produce). Anyway, this reminds me of an insight I had several years ago which I used to tell the students in my decision analysis classes. They were always skeptical but maybe now with research behind me on this, they'll believe me.

Anyway, here goes:

When I was younger, people used to complain about candy bars getting smaller and smaller. (For example, Stephen Jay Gould has a graph in one of his books showing the size of the standard Hershey bar declining from 2 ounces in 1965 gradually down to 1.2 ounces in 1980, and for that matter I can recall tunafish cans gradually declining from 8 ounces to 6 ounces.) And I remember going to the candy machine with my quarter and picking out the candy bar that was heaviest--I don't remember which one--even if it wasn't my favorite flavor, to get the most value for the money.

But now I realize that, rationally, candymakers should charge more for smaller candy bars. The joy from eating the candy is basically discrete--I'll get essentially no more joy from a 1.7-ounce bar than from a 1.4-ounce bar. But the larger bar will be worse for my health (no big deal if I eat just one, but with some cumulative effect if I eat one every day, similarly with the sodas and so forth). And, given the well-known fact that nobody can eat just part of a candy bar, I get more net utility from the small bar, thus they should charge more.

Grazia passed on this link to a report by Joel Waldfogel:

People with higher incomes today report higher levels of happiness than their poorer contemporaries. At the same time, people today are far richer than earlier generations, but they're not happier than those who came before them. In light of such wrinkles, a growing cadre of economists has cut out the money middleman and moved to studying happiness directly. The latest installment in this genre is a new study by economists David Blanchflower of Dartmouth and Andrew Oswald of Warwick. They document how happiness evolves as people age. While income and wealth tend to rise steadily over the life cycle, peaking around retirement, happiness follows a U-shaped age pattern.

It's a good news article, with data details:

Kim Jinnett writes,

I wrote the following (with Aleks Jakulin) to introduce a special issue on Bayesian statistics of the journal Statistica Sinica (volume 17, 422-426). I think the article might be of interest even to dabblers in Bayes, as I try to make explicit some of the political or quasi-political attitudes floating around the world of statistical methodology.

Bob O'Hara has a new blog with stuff on OpenBugs (see here). Feel free to ask him when they're going to set up the new OpenBugs so it can work with R2WinBUGS!

Aleks forwarded this article by Danah Boyd:

Over the last six months, I [Boyd] have noticed an increasing number of press articles about how high school teens are leaving MySpace for Facebook. That's only partially true. There is indeed a change taking place, but it's not a shift so much as a fragmentation. Until recently, American teenagers were flocking to MySpace. The picture is now being blurred. Some teens are flocking to MySpace. And some teens are flocking to Facebook. Who goes where gets kinda sticky... probably because it seems to primarily have to do with socio-economic class.

Much discussion follows. No numbers but I imagine somebody's done that too.

A few days ago we had quite a discussion on multidimensional scaling. While everyone agreed that initialization is important with non-convex problems, minimizing some objective function is more appealing than using initial placement for the prior, except in appealing circumstances such as iterative scaling. For the objective function approach, one can regularize the stress function, and it is also possible to use the prior to shrink towards geographic positions.

The untidy initial placement approach is sufficient, however, to provide a visualization as we travel from the initial placement towards the final placement. Namely, the clinal pattern in the final placement is only one of the things we can learn: the migrations of points and the resulting stresses are just as interesting in providing insight about the differences between the simple uniform geographic diffusion model and the real distribution of genes in Europe.

These are cool. (Also in blog form, apparently.).

I noticed this report of a new study on birth order and IQ:

The eldest children in families tend to develop higher I.Q.’s than their siblings, researchers are reporting today, in a large study that could settle more than a half-century of scientific debate about the relationship between I.Q. and birth order.

The average difference in I.Q. was slight — three points higher in the eldest child than in the closest sibling — but significant, the researchers said. And they said the results made it clear that it was due to family dynamics, not to biological factors like prenatal environment.

Here's more from Petter Kristensen, coauthor of the study:

Seth and I are both pretty aggressive conversationalists in person. But on the web, we're much different. I have an equanimous, mellow blog personality and generally try to see both sides of an issue. Seth is more of a tough guy; see, for example, here and (amazingly) here.

I can think of a couple reasons for this. First, Seth and I have different incentives. As a bestselling author, he has much to gain and little to lose by making provocative statements. In contrast, as a primarily academic writer, I think I can lose more than I gain in reputation by being controversial (except, of course, where I can back up my ideas with research, but even there I try to use a mellow tone; see, for example, here and here).

Another explanation has to do with the norms of different academic fields. Statisticians are trained to say No a lot and worry about uncertainty; in contrast, Seth has been doing a lot of self-experimentation which has given him more of an appreciation for bold conjecture.

The driver overestimates his control over the situation (including his own car as well as others on the road). The backseat driver ("Whoa--you're taking that curve too fast!") underestimates the driver's control. As a driver, I listen to the passengers because they provide a useful corrective. Even if the backseat driver is sometimes annoying, it makes sense to listen.

More generally: I'll take anybody's advice seriously.

Yesterday we were looking at the musical taste proximity between European countries. But what about the proximity between European nations in terms of the genes? The field of population genetics investigates this problem. I have taken some Y chromosome data, and computed the distance between two nations based on their genetic distance.

The result, obtained with MDS is as follows:

I've color-coded different language groups. We can see that North Africans are quite different, and that within Europe, there is a clear gradient from the East to West, with several clusters. The islands of Gotland and Sardinia are composed of a diverse mix from different populations.

The interesting point, however, is that I've initialized the positions of points to the geographic positions, which can roughly be interpreted as a prior. This is a bit unusual: usually the points are randomly initialized, or initialized with some sort of a linear dimension reduction technique, such as with Torgerson's procedure.

We came across a interesting paper on missing data by Nicholas J. Horton and Ken P. Kleinman. The paper is about comparison of Statistical Methods and related Software to Fit Incomplete Data Regression Models.

Missing data are a recurring problem that can cause bias or lead to inefficient analyses. Statistical methods to address missingness have been actively pursued in recent years, including imputation, likelihood, and weighting approaches. Each approach is more complicated when there are many patterns of missing values, or when both categorical and continuous random variables are involved. Implementations of routines to incorporate observations with incomplete variables in regression models are now widely available. We review these routines in the context of a motivating example from a large health services research dataset. While there are still limitations to the current implementations, and additional efforts are required of the analyst, it is feasible to incorporate partially observed values, and these methods should be used in practice.

Jouni pointed me to this forthcoming book by Jim Albert. Here are the table of contents:

An introduction to R.- Introduction to Bayesian thinking.- Single parameter models.- Multiparameter models.- Introduction to Bayesian computation.- Markov chain Monte Carlo methods.- Hierarchical modeling.- Model comparision.- Regression models.- Gibbs sampling.- Using R to interface with WinBUGS.

At 280 pages, Jim's book looks like it will be a great place for people to get started.

I'll also recommend Appendix C of BDA, where we get you started and work through a basic hierarchical model in R/Bugs and then program it in R alone. In doing these, we work through different parameterizations of the model.

P.S. In the first version of this blog entry I'd judged the contents of the book too quickly. Jim Albert sent me more info:

Robin Hanson points out that biological systems that have a useful function are not necessarily optimal when put in new environments. This reminds me of an interesting interesting article by Witold Rybczynski where I learned that the structural engineer Ove Arup agreed:

...The idea that the correct functional, the correct structural and the best possible aesthetic solutions are one and the same thing must, I am afraid, be abandoned together with the older philosophers' dream about the harmony and ultimate identity of truth, goodness, justice and beauty.

For example, Orup wrote:

A wall like the one at Highpoint would have been cheaper to build with bricks, but [Lubetkin] claimed it was functional and economic. It wasn't functional at all: it had to be "Modern." Functionalism really became a farce. What is wrong with a sloping roof? They can't afford to pay what it costs to make a flat roof really waterproof. Lubetkin didn't care. He just cared for the picture in the architectural magazines.

Igor Carron reports on a paper by Richard Barniuk and Michael Wakin, "Random projections of smooth manifolds," that is billed as a universal dimension reduction tool. That sounds pretty good.

I'm skeptical about the next part, though, as described by Carron, a method for discovering the dimension of a manifold. This is an ok mathematical problem, but in the problems I work on (mostly social and environmental statistics), the true dimension of these manifolds is infinity, so there's nothing to discover. Rather than a statement like, "'We've discovered that congressional roll-call votes fall on a 3-dimensional space" or "We estimate the dimensionality of roll-call voting to be 3" or even "We estimate the dimensionality to be 3 +/- 2", I prefer a statement like, "We can explain 90% of the variance with three dimensions" or "We can correctly predict 93% of the votes using a three-dimensional model" or whatever.

Dan Goldstein quotes J. Scott Armstrong:

About two years ago, I [Armstrong] was a reasonable person who argued that tests of statistical significance were useful in some limited situations. After completing research for “Significance tests harm progress in forecasting” in the International Journal of Forecasting, 23 (2007), 321-327, I have concluded that tests of statistical significance should never be used.

Bob Shackleton writes,

G. K. Chesterton writes, at the end of his celebrated book on George Bernard Shaw:

I know it is all very strange. From the height of eight hundred years ago, or of eight hundred years hence, our age must look incredibly odd. We call the twelfth century ascetic. We call our own time hedonist and full of praise and pleasure. But in the ascetic age the love of life was evident and enormous, so that it had to be restrained. In a hedonist age pleasure has always sunk low, so that it had to be encouraged. How high the sea of human happiness rose in the Middle Ages, we now only know by the colossal walls that that they built to keep it in bounds. How low human happiness sank in the twentieth century our children will only know by these extraordinary modern books, which tell people that it is a duty to be cheerful and that life is not so bad after all. Humanity never produces optimists till it has ceased to produce happy men. It is strange to be obliged to impose a holiday like a fast, and to drive men to a banquet with spears. But this shall be written of our time: that when the spirit who denies beseiged the last citadel, blaspheming life itself, there were some, there was one especially, whose voice was heard and whose spear was never broken.

Chesterton was a Catholic conservative of the early 1900s, Shaw was a socialist, and both were famous for expressing their ideas in paradox.

Shaw, the leftist, associated progress with material happiness, while Chesterton, the rightist, said things were better in the Middle Ages. Nowadays, the debates usually go in the other directions, with people on the left being less positive about material progress and people on the right saying that things are great now and are getting better. (See, for example, Will Wilkinson's skeptical take on happiness
research.)

I don't have anything to add here except to note the interesting switch of polarity, which reminds me of my thoughts here and here on the changing views of left and right regarding science.

Shravan Vasishth writes,

One thing I [Shravan] have not understood so far (and I'm on the 250th page or so [of Data Analysis Using...]) is what the fully Bayesian approach buys me as an experimenter. As far as I can tell, the major advantage is that I can leverage previous related experiments to set up an "informative" prior. But it's not clear yet what this leveraging will give me that the conventional approach won't. I understand and appreciate the importance of the simulation idea, I do that all the time for computational modeling of sentence processing, but the major gains from the Bayesian approach for data analysis are not clear to me yet.

My reply: Bayes gives you partial pooling using a convenient formula (see, for example, chapter 5 of Bayesian Data Analysis, or the equivalent chapters in Carlin and Louis or other books). You can do partial pooling "manually" but that's more work and gets tougher for models with varying intercepts and slopes, and for nonnested models. Also, by being based on a generative model, Bayesian inferences can be checked by comparing data to replications simulated under the model.

Russell writes:

Have you ever taken a look at the version numbers of the packages in CRAN. The median and probably the third quartile are below 1.0. What does that imply about statisticians? R programmers?

I say, let's just start with version 2.0 and go from there; that will relieve some of the pressure to get version 1 working ok.

How to talk so kids will listen and listen so kids will talk, by Adele Faber and Elaine Mazlish. I read this book long before I had kids--it's incredibly helpful for interactions with adults as well. It's definitely #1 on my list: a book that really changed my life.

How animals work, by Knut Schmidt-Nielsen. What can I say--this is cool stuff. Physics really works.

Low life, by Luc Sante. This one needs no introduction, I think. I happened to read it around when it came out--I'm not sure how I encountered it--and what struck me was its utter deadpan tone.

The honest rainmaker, by A. J. Liebling. I'm a sucker for this kind of stuff. Although it should probably be classified as fiction.

Life on the Mississippi, by Mark Twain. OK, it's basically fiction too, also has some dead spots, but still has a great treatment of one of my favorite themes, which is that so much that seems permanent is not.

The last laugh, by Phil Berger. Doesn't really belong in this list--it's good, but probably not great--but it's my favorite of the books I've finished lately.

That reminds me, by Tony Randall. No kidding. The only way this book could be improved would be to have some indicator of which of the stories are actually true.

Plagues and peoples, by William McNeill. Understanding history in terms of micro- and macro-parasitism. May seem obvious now, maybe could be updated, but as far as I know was pretty trailblazing. Full of fun facts.

The origins of the second world war, by A.J.P. Taylor. Compulsively readable, also seems (to this non-expert) to be full of insight.

Baseball's greatest quotations, by Paul Dickson. OK, not really in the top 200 even, but I wanted to include something by this charming nonfiction writer.

Ball four, by Jim Bouton. This one really does belong on the list. (I'd also like to include the Bill James abstracts, but I'm not sure which ones to pick, and they have weaknesses as well as strengths. I'll save them for my list of the best statistics books ever.)

The death and life of great American cities, by Jane Jacobs. OK, another classic--sorry, the list is getting less and less idiosyncratic.

Judgment under uncertainty: heuristics and biases, edited by Daniel Kahneman, Paul Slovic, and Amos Tversky. An amazingly good book with an incredible quality level, especially considering it's an edited volume.

The great war and modern memory, by Paul Fussell. After this one he got crankier and crankier, but this one is essential.

Also I have a soft spot for my own books, but due to lack of critical distance I'll keep them off this list.

One of the small puzzles of decision analysis is that:

(a) Plans have lots of problems--things commonly don't go according to plan, plans notoriously exclude key possibilities that the planner didn't think of, plans can encourage tunnel vision, etc. But . . .

(b) Plans are helpful. In fact, it's hard to do much of anything useful without a plan. (I'm sure people will come up with counterexamples here, but certainly in my own work and life, not much happens if I don't plan it. Serentipitous encounters are fine but don't add up to much.

Beyond this, one could add that economic activity seems to work well with minimal planning (just enough structure and rules to set up "the marketplace") but individual actors plan, and need to plan, all the time.

This puzzle is particularly interesting to me as I do work in applied decision analysis.

So what's the solution to the puzzle?

Leonardo Monasterio sent me this link. I'd give my thoughts but I don't have much to add beyond what's already in that blog entry. I wouldn't consider this a serious statistical tool but it's amusing.

Actually, this whole Strange Maps blog is cool.

I was having an interesting discussion with Seth about his claim that "the overall benefits of health care are probably minor." The basis of his claim is evidence cited by Aaron Swartz:

In the 1970s, the RAND Corporation picked out 7700 people in six cities and gave half of them free health care. Those lucky ones took advantage of it (spending 30-40% more on average) and they spent it on reasonable things (as judged by medical observers), but they didn’t seem to get any healthier. . . . The RAND study was by far the biggest study of this kind, but other studies find similar results. One analysis found that regions whose Medicare programs give out more money (when the underlying healthiness of the residents is held constant) see no increase in survival rates. A replication found the same results in VA hospitals. Cross-national comparisons find “the impact of public spending on health is … both numerically small and statistically insignificant”. Correlational studies find “Environmental variables are far more important than medical care.” And there are more where that came from.

Several discussants (including myself) at Seth's blog were skeptical about his skeptism, citing various successful medical treatments (in my case, fixing a compound fracture of the wrist; others mentioned cancer treatment, etc.). Seth responded:

The RAND study, of course, is limited — but is there a better attempt to figure out the overall value of medicine? I don’t know of one. if you can point me to a study that shows the more-than-minor value of modern medicine I’d love to look at it. . . . when the overall effectiveness of medicine has come under scrutiny, it has not fared well — and the RAND study is a good example.

Total vs. marginal effects

I have not looked at the Rand study so can't comment on the details, but my first thought is that the marginal benefits from additional health care will be less than the benefits from good existing care. So, even if more is not much better, that doesn’t mean that the overall benefits of existing care are “minor.”

From a policy standpoint, it is the marginal effects that are the most interesting, since nobody (even Seth?) is proposing to zero out medical treatment. Presumably there are diminishing returns, and the benefit/cost ratio for additional treatment is less than that for existing treatment. (And, indeed, some medical care can make things worse, even in expected value; for example, you can get catch the flu in the doctor's waiting room.) But, unless I'm missing something, Seth and Aaron are confusing marginal with total effects.

P.S. Also see Robin Hanson's discussion (with lots of links), which explicitly distinguishes between marginal and total effects. Here I'm not expressing any position on the marginal effects of health care (given my ignorance on the topic), just pointing out that Robin's position seems to have become overstated by others.

P.P.S. See Jake Bowers's comments below. Also more discussion here.

I just received the following email:

Dear Andy,

Please disregard my previous email. I found already a clustering algorithm that works for us. . . .

Like I always say, one of the best things a statistician can do is just stand aside and let people do their work.

Etienne Rivot sends in a question about models for missing data. The issues are subtle and I think could be of general interest (since we all have missing data!) These issues are covered in Chapter 7 of Bayesian Data Analysis, but it always helps to see these theoretical ideas in the context of a specific example.

Rivot writes:

In recent years there has been a lot of discussion of polarization in American politics, and also some scholarly debate, with some researchers finding polarization (most notably, Joe Bafumi and Bob Shapiro's "stubborn American voter") and others (notably, DiMaggio et al.) finding stability in issue attitudes. The best synthesis, I think, is the book "Culture Wars? The Myth of Polarized America" by Fiorina, Abrams, and Pope, who find polarization in attitudes toward the Democrats and Republicans, but explain this polarization as a consequence of the parties' positions rather than extremism in voters' issue attitudes.

That's the background to this paper by Delia Baldassarri and myself, and here's the abstract:

Political polarization is commonly measured using the variation of responses on an individual issue in the population: more variation corresponds to more people on the extremes and fewer in the middle. By this measure, research has shown that--despite many commentators' concerns about increased polarization in recent decades--Americans' attitudes have become no more variable over the past two or three decades. What seems to have changed is the level of partisanship of the electorate.

We define a new measure of political polarization as increased correlations in issue attitudes and we distinguish between issue partisanship--the correlation of issue attitudes with party ID and liberal-conservative ideology--and issue alignment--the correlation between pairs of issues. Using the National Election Studies, we find issue alignment to have increased within and between issue domains, but by only a small amount (approximately 2 percentage points in correlation per decade). Issue partisanship has increased more than twice as fast, thus suggesting that increased partisanship is not due to higher ideological coherence. Rather, it is parties that are more polarized and therefore better at sorting individuals along ideological lines; the change in people's attitudes corresponds more to a re-sorting of party labels among voters than to greater constraint on issue attitudes.

We conclude suggesting that increased issue partisanship, in a context of persistently low issue constraint, might give greater voice to political extremists and single-issue advocates, and amplify dynamics of unequal representation.

Jacob Felson writes,

From J. Robert Lennon's rant, I learn about an article by Mark Helprin advocating unlimited copyright. Lennon writes:

It seems he [Helprin] would like copyrights to extend forever, thus allowing Disney to get rich off its stale creations for eternity. Here, though, is the money quote:

Were I tomorrow to write the great American novel (again?), 70 years after my death the rights to it, though taxed at inheritance, would be stripped from my children and grandchildren.

Can you see the mistake [writes Lennon]? No, no, not the parenthetical "again?", which is almost too pathetic to mention. The mistake is that the rights to his imaginary masterpiece would not be "stripped" from his heirs--in fact, his heirs would keep all their rights. They would just have to share them with everybody else.

I'm pretty sure that in 70 years my descendants will have more important things on their minds than royalty checks for Bayesian Data Analysis, so I can't say I have a dog in this fight--what interests me here is the political angle.

Helprin is well known as one of the few literary-type writers with strong Republican sympathies. (I'm sure there are others . . . well, obviously there's Tom Wolfe, and John Updike wrote about his support for the Vietnam War, hmm, Evelyn Waugh would be a Republican if he were living now, right? And P.J. O'Rourke isn't really a literary-type writer but surely he has the ability to do so if he were to put his mind to it . . . then there's Christopher Buckley, but does he really count? Maybe that Deliverance guy was pretty conservative, I dunno--hard to tell the politics from the novel. Kingsley Amis. And there's Kipling but that takes us back a bit. Anyway, you get the idea.)

I don't know anything about the background of Helprin's interest in copyright law, but I wonder whether his opinion came partly in reaction to the fact that leftist anti-corporate types are the ones opposing copyright extensions. (This is the chicken-egg question referred to by my title above.) Just as, in reverse, a Democrat might oppose the death penalty in reaction to the people who are its most visible supporters. It's an instinctive pro-business stance or an instinctive anti-business stance; either can be appropriate in any specific situation but either represents an "ideology" in the sense of being a perspective from which each specific situation is viewed.

In our recent research, Delia and I have found surprisingly low correlations between issue attitudes--but, to the extent these correlations are increasing, one possible explanation is that people are starting to align their attitudes with the attitudes of their allies (and against those of their enemies).

I heard from David Feitler, who had sent this. He writes:

Nisha Gottfredson writes:

Jacob Felson writes,

Jarrett Byrnes writes,

A group of us are working through your Multilevel book, and a question has come up regarding models incorporating multiple predictors. We were working some of the chapters on using simulation to draw inference, but have been been puzzling over how one can represent their data with fitted and simulated lines, one factor at a time. True, one can show the fitted and simulated model for a variety of other factors that are not of interest, but this seems unsatisfactory, particularly if you are incorporating three or more predictors on your model. Do you have any suggestions as to how one can best present data and models for these more complex models, such that a reader can assess the relationship between the model and the data for each single predictor?

My reply: I think you have two questions:

1. How to display a fitted regression that has many input variables?

2. How to display a sequence of fitted models?

I'll discuss each issue in turn. (Warning: for neither problem do I have a great answer yet.)

1. How to display a fitted regression that has many input variables? I'd start with curves of the expected value of y as a function of each input, using a separate plot for each input variable and multiple curves as necessary to show interactions. See, for example, the graphs on page 91 of our new book. With more than two inputs, I'd probably stack the graphs vertically. We're still working on our general R function for this. And I'd also display the estimated coefficients (as in the lower graph on page 337 of our book), probably after standardizing the inputs by subtracting means and dividing by two standard deviations.

When a model has three-way interactions or many two-way interactions, the displays start to get tricky, and I have no great answer yet. I do think, though, that if we try harder we'll gradually make progress here. Traditionally, graphical methods have focused on displaying raw data; as that same ingenuity is used to display inferences and fitted models, I think good new general plots will arise.

2. How to display a sequence of fitted models? This is a really important question, and again I don't see anything great right now. The series of graphs for the arsenic example in chapter 5 of our book give some sense of what can be done, but we're pretty disorganized there. It would be good to have a coherent display to visualize what happens to a model when a predictor is added, something like the graphs on page 12 of this paper. I will add, though, that I am not particularly interested in model selection or model averaging, at least as these concepts are typically formulated statistically. I'm more interested in putting together a good model and using simpler models as steps in understanding what the ultimate model is doing.

My sister and her family visited recently. She told me that now when her little kid brushes his teeth, he pretends that his lower teeth are the #1 subway track, and the upper teeth are the #7 subway track (and his tongue is the sidewalk).

Hey, this looks interesting:

When you run glm in R, the default is linear regression. To do logistic, you need a slightly awkward syntax, e.g.,

fit.1 <- glm (vote ~ income, family=binomial(link="logit"))

This isn't so bad, but it's a bit of a pain when doing routine logistic regressions. We should alter glm() so that it first parses the y-variable and then, by default, chooses logit if it's binary, linear if it's continuous, and overdispersed Poisson if it's nonnegative integers with more than two categories, maybe ordered logit for some other cases. If you don't like defaults, you can always specify the model explicitly (as you currently must do anyway).

Aliasing the standard glm() function is awkward so we'll start by putting these defaults into bayesglm() (which I think we should also call bglm() for ease of typing) in the "arm" package in R. The categorization of the variables fits in well with what we're doing in the "mi" package for missing-data imputation.

OK, I think you know the answer I want youall to give to the above question . . . anyway, I got the following email from someone who will be an assistant professor teaching econometrics to masters students:

I have not yet decided on a textbook. I've been reviewing books like Stock and Watson, though, and Greene's econometrics textbook, but I'm undecided. I purchased your book with Jennifer Hill the other day, and absolutely love it. I love how readable it is, and how practical it is in its orientation, but at the same time, how rigorous it is. Ordinarily, I am selecting among textbooks that are practical and readable but not technical, or (as is usually the case) technical but not aimed towards the practitioner and hardly readable. That said, I was wondering that since I'm not finished with the book, whether you could advise me about the appropriateness of your book for a masters level econometrics course in an economics department?

My quick answer is, Yeah, I think it would be excellent for an econometrics class if the students have applied interests. Probably I'd just go through chapter 10 (regression, logistic regression, glm, causal inference), with the later parts being optional.

But what do others think? My book does have a blurb from an economist, but is there essential info for an econometrics class that's not in our book?

P.S. to readers: I'm usually pretty good at trimming the gratuitous compliments from these emails--I kept them in here only because they are, as they say in the movies, essential to the plot.

This isn't a particularly original thought, but . . . I was reading this discussion by Rhian Ellis about fiction and nonfiction:

Fiction teaches some excellent, if subtle, lessons. There's the basic one: Other people have some of the same thoughts and feelings as you do, and its relative: Other people have lives that are incredibly different from yours.

and this reminded me of something I realized awhile ago, I guess around the time I started writing books, which is that (most) fiction is about reality. To expand on this, when I was a kid and read lots of books, I thought of fiction as stories that people make up. But my impression now is that many of the best stories are true--basically, I'm talking about anecdotes as well as longer stories that might be categorized as "gossip" or "life stories"--and that fiction is a way to try out various possibilities, move things around, do various what-ifs.

My books are all nonfiction (I hope) but they do have some narratives in them, and I think the key for me was realizing that I wanted to write because I had things I want to tell people (including myself). Which is slightly different from the other motivation for storytelling, the motivation I knew about when I was a kid, which is to entertain people. The paradox (but it isn't really, no paradox is really a paradox once you open it up) is that the best entertainment can come from stories that someone really feels he or she needs to tell.

Hal Daume writes,

I hope you don't mind an unsolicited question about your book. I'm working with someone in our chemistry department right now on a regression problem, and he's a bit worried about the "normality of errors" assumption. You state (p.46):

The regression assumption that is generally the least important is that the errors are normally distributed... Thus, in contrast to many regression textbooks, we do not recommend diagnostics of the normality of regression residuals.

Can you elaborate on this? In particular, what if the true error distribution is heavy-tailed? Could this not cause (significant?) problems for the regression? Do you have any references that support this claim?

My response: It depends what you want to do with the model. If you're making predictions, the error model is certainly important. If you're estimating regression coefficients, the error distribution is less important since it averages out in the least-squares estimate. The larger point is that nonadditivity and nonlinearity are big deals because they change how we think about the model, whereas the error term is usually not so important. At least that's the way it's gone in the examples I've worked on.

To get slightly more specific, when modeling elections I'll occasionally see large outliers, perhaps explained by scandals, midterm redistrictings, or other events not included in my model. I recognize that my intervals for point predictions from the normal regression model will be a little too narrow, but this hasn't been a big concern for me. I'd have to know more about your chemistry example to see how I'd think about the error distribution there.

Masanao suggests this door for the Playroom:

I found this interesting article on information software and interaction.

Here is the abstract:

The ubiquity of frustrating, unhelpful software interfaces has motivated decades of research into “Human-Computer Interaction.” In this paper, I suggest that the long-standing focus on “interaction” may be misguided. For a majority subset of software, called “information software,” I argue that interactivity is actually a curse for users and a crutch for designers, and users’ goals can be better satisfied through other means.

Information software design can be seen as the design of context-sensitive information graphics. I demonstrate the crucial role of information graphic design, and present three approaches to context-sensitivity, of which interactivity is the last resort. After discussing the cultural changes necessary for these design ideas to take root, I address their implementation. I outline a tool which may allow designers to create data-dependent graphics with no engineering assistance, and also outline a platform which may allow an unprecedented level of implicit context-sharing between independent programs. I conclude by asserting that the principles of information software design will become critical as technology improves.

I'm not a pie chart person. But here is an example where I don't mind the use (I found it here):

Using a list of countries generated by The World Factbook database, flags of countries fetched from Wikipedia (as of 26th May 2007) are analysed by a custom made python script to calculate the proportions of colours on each of them. That is then translated on to a piechart using another python script. The proportions of colours on all unique flags are used to finally generate a piechart of proportions of colours for all the flags combined. (note: Colours making up less than 1% may not appear)

Duncan Watts wrote an op-ed in NY Times on The Politics of Eurovision. There he writes:

I had heard about this practice, of course, whereby geographical and cultural neighbors tend to vote for each other, and nobody votes for Britain (well, except for Malta). But it was startling to see just how flagrant it was. The Scandinavians all voted for one another; Lithuania gave 10 points to Latvia (whose entry, bizarrely, sang in Italian); former Warsaw Pact countries voted for Russia; and almost nobody voted for Britain (surprisingly, Ireland did — and, of course, Malta).

Eurovision has been studied by academics a couple of times by now: Derek Gatherer titled his paper "Comparison of Eurovision Song Contest Simulation with Actual Results Reveals Shifting Patterns of Collusive Voting Alliances" and Anthony Dekker has a paper The Eurovision Song Contest as a 'Friendship' Network. The titles are not very forgiving, and here is an example of a chart from one of these two papers (neither of which has been authored by Duncan, of course):

(there isn't a single Baltic country in the area denoted as "Baltic", but they are all in the Balkan peninsula).

Serbs and Albanians are pretty much in a state of war, yet they seem to be aligned together in a "friendship" network? The same goes for Greece and Turkey in 2005, and many other similar pairs. I guess the young form a voting alliance for the preservation of hip-hop and metal, and the old too form into an opera friendship. It's not preferences - it's politics! It's not musical taste, it's alliances! It's not quality, it's who is friends with whom! It's not the fact that musicians in Britain, Ireland, France, who top the charts, already have an excellent means of commercially deploying their music beyond the confines of their country. It's rare for a first-class musician from the West to try for Eurovision: why expose themselves to public ranking and ridicule in case of failure if they can already sell lots of music by other means. But not so with the East; for the musicians there, Eurovision is the best way of going beyond their borders.

It's true that Balkan music is overrepresented in the voting scores: it has many small countries with similar musical taste and little population. But as this Excel spreadsheet shows, Serbia would win even if all other countries were prevented from voting. In case only the 1994 members were allowed to vote, Turkey would win instead, followed by Serbia in the second place. The results are quite robust with respect to countries that are allowed to vote. If, however, we weighed the votes by population, allowed countries to vote for themselves, and excluded non-1994EU, the winning order would be Turkey, Greece, Serbia. So, even by population-weighting the votes, the results do not change much.

There are some interesting nonlinearities. It's known that novelty value plays a big role at Eurovision: and there will be little novelty value to British, Irish, Spanish and German music that are so successful and ubiquitous in the marketplace: they won't get novelty scores like some more exotic types of music will: the past winners include goofy entries such as Finnish monsters, Ukrainian warriors and an Israeli transvestite. On the other hand, anyone who has tried listening to classical music knows that it takes some exposure before you can enjoy music, just as it takes a certain amount of exposure to literature to enjoy poetry.

The reaction from Britain was quite harsh, dismissing scoring as pure politics. But the UK song should be examined in the context of the 1990 winner on a similar theme. Google can translate the lyrics and you can compare them to the UK entry. Among other events, that 1990 song was what inspired the chain of events that led to the break-up of Yugoslavia (where some of the states were pro-EU and others against), and possibly other nations too. For musical quality, compare it to the UK 1997 winner.

So, in summary, there are many models that explain correlations in data. A cluster in votes can be interpreted both as an alliance to win the majority, or it can be equivalently interpreted as a group of countries that shares cultural preferences. One interpretation is cynical, the other is respectful. If you are an author that doesn't distinguish between the Balkans and Baltic, you might find it hard to decide on the right interpretation. A respectful one is a safer bet.

Stan Salthe pointed me to the article "Why Mathematical Models Just Don't Add Up." It's something every quantitative modeler should read. Let me provide some snippets:

The predictions about artificial beaches involve obviously absurd models. The reason coastal engineers and geologists go through the futile exercise is that the federal government will not authorize the corps to build beaches without a calculation of the cost-benefit ratio, and that requires a prediction of the beaches' durability. Although the model has no discernible basis in reality, it continues to be cited around the world because no other model even attempts to answer that important question. [...] In spite of the fact that qualitative models produce better results, our society as a whole remains overconfident about quantitative modeling. [...] We suggest applying the embarrassment test. If it would be embarrassing to state out loud a simplified version of a model's parameters or processes, then the model cannot accurately portray the process. [...] A scientist who stated those assumptions in a public lecture would be hooted off the podium. But buried deep within a model, such absurdities are considered valid.

While the article is quite extreme in its derision of quantitative models, plugs the book the authors wrote, and employs easy rhetoric by providing only positive examples of a few failures and not negative examples of many successes, it is right that quantitative models are overrated in our society, especially in domains that involve complex systems. The myriad of unrealistic and often silly assumptions are hidden beneath layers of obtuse mathematics.

Statistics and probability were attempts to deal with failure of deterministic mathematical models, and Bayesian statistics is a further attempt to manage the uncertainty arising from not knowing what the right model is. Moreover, a vague posterior is a clear signal that you don't have enough data to make predictions.

Someone once derided philosophers by saying that first they stir up the dust, and then they complain that they cannot see: they are taking too many things into consideration, and this prevents them from coming up with a working model that will predict anything. One does have to simplify to make any prediction, and philosophers are good at criticizing the simplifications. Finally, even false models are known to yield good results, as we are reminded by that old joke:

Steven Pinker writes,

Though we live in an era of stunning scientific understanding, all too often the average educated person will have none of it. People who would sneer at the vulgarian who has never read Virginia Woolf will insouciantly boast of their ignorance of basic physics.

This statement of the form, "if A, then B," is hard to evaluate since, strictly speaking, it's true if A is the empty set. Will "the average educated person" reallly sneer at the vulgarian etc? All the time I'm sneering at the vulgarians who use tables instead of graphs, but, hey, that's my job. I sneer at the people who sneer at the people who sneer at the vulgarians.

In a comment to this entry on Bayesian exploratory data analysis, Carolyn asks why I love the idea so much.

My response: People will often not use a statistical method until it has a theoretical justification. In particular, many Bayesians don't seem to want to check their models. This is something that I noticed, and that disturbed me, at the 1991 Valencia meeting. Graphs of raw data, graphs of posterior inferences, graphs of convergence of Gibbs samplers--but no plots comparing data to simulated replicated data. Even though this seemed (to me) like such a natural idea, and it had been done (in a non-Bayesian context) by distinguished statisticians including Mosteller in 1954 and Ripley in 1987.

How could it be that Bayesians--the most model-bound statisticians of all--were doing less model checking than classical statisticians? Worse, I found many Bayesians to be actively hostile to model checking. I had an example with 20,000 parameters where the chi^2 statistic was 30,000, and Bayesians were telling me I wasn't allowed to say this was a problem with the model.

My take on this was that these Bayesians needed a theoretical justification for model checking, a theoretical framework under which model checking is a natural part. We did a bit of this in our earlier paper (Gelman, Meng, and Stern, 1996). My 2003 paper (linked to in the above blog entry) put this in the context of other generalizations of Bayesian theory such as hierarchical modeling and missing-data modeling (see Section 2.1 of that paper) and put exploratory data analysis in the same theoretical framework. EDA and modeling are usually considered opposites, so I really liked the idea of putting them together.

I think I've succeeded quite a bit (although not completely) in the mathematics, but I still have a lot to go in the sociology, in that most Bayesians still don't use the y.rep notation that I think is key to understanding the application and checking of models. I'm hoping that the next paper, based on ideas in Jouni's thesis, will push things along.

Stephen Dubner writes about a professor of economics who makes zillions of dollars consulting. I'm not surprised by this, because my impression is that legal consulting is an extremely inefficient market. In several of the cases I've consulted on, the statistical expert (typically not actually a statistician) has been truly incompetent, in one case not being certified as an expert in the relevant area by the court, and in another case--which was truly memorable--dividing by N (the population size) rather than n (the sample size) in computing the estimation variance from a random sample. (I was really looking forward to the exchange with this guy in the courtroom--I mean, what could he say, he's a sampling expert and divided by the wrong N?--but, like most cases, it got settled before reaching court.)

I can give other stories, but the key point is that lawyers hire incompetent statistical experts, even in cases that are important. It's gotta be worth their money to hire better consultants, but presumably they can't find them. Actually, I think that I've probably cost less than the opposing consultant in every case I've worked on, since, despite my high hourly rate, I'm trying to minimize my consulting hours, whereas I imagine that professional consultants are, if not trying to maximize hours, at least to keep their business going. But most clients don't know to hire me (or the equivalent)--I think they pretty much get their consultants by word of mouth or through some casual search. (I still can't figure out why the Gore team in the 2000 election hired a statistical consultant who had, as far as I know, never worked seriously on election data before, given that there are so many quantitative political scientists out there. (I don't actually know who the Bush team hired, but since they won, I guess they got their retrospective money's worth.))

So, anyway, to get back to the econ professor guy: it's probably worth clients' money to hire this guy--I imagine his team has a minimum level of professionalism that's much better than what's usually out there. Given the high stakes in many legal cases, and the relative simplicity of the statistical questions that arise, I'm surprised that clients can't do a better job in finding competent statistical experts.

Francesca Vandrola writes,

I love this paper. Here's the abstract (yes, it's too long, I know):

Exploratory data analysis (EDA) and Bayesian inference (or, more generally, complex statistical modeling)--which are generally considered as unrelated statistical paradigms--can be particularly effective in combination. In this paper, we present a Bayesian framework for EDA based on posterior predictive checks. We explain how posterior predictive simulations can be used to create reference distributions for EDA graphs, and how this approach resolves some theoretical problems in Bayesian data analysis. We show how the generalization of Bayesian inference to include replicated data y.rep and replicated parameters theta.rep follows a long tradition of generalizations in Bayesian theory.

On the theoretical level, we present a predictive Bayesian formulation of goodness-of-fit testing, distinguishing between p-values (posterior probabilities that specified antisymmetric discrepancy measures will exceed 0) and u-values (data summaries with uniform sampling distributions). We explain that p-values, unlike u-values, are Bayesian probability statements in that they condition on observed data.

Having reviewed the general theoretical framework, we discuss the implications for statistical graphics and exploratory data analysis, with the goal being to unify exploratory data analysis with more formal statistical methods based on probability models. We interpret various graphical displays as posterior predictive checks and discuss how Bayesian inference can be used to determine reference distributions.

We conclude with a discussion of the implications for practical Bayesian inference. In particular, we anticipate that Bayesian software can be generalized to draw simulations of replicated data and parameters from their posterior predictive distribution, and these can in turn be used to calibrate EDA graphs.

Also this paper.

Aleks pointed me to this. Usually I'm not such a fan of tricky displays, but I have to admit that this one is kind of pretty.

We'll be doing the Playroom all summer, Wed and Thu afternoons from 12-5 in 707 International Affairs Bldg. The rough schedule is: statistics on Wednesdays, political science on Thursdays, but you can come by either day on either topic; people will be coming in and out, working on things, and meeting in small groups. Each Wed 2pm we're having our weekly meeting on multilevel modeling, causal inference, and missing data with Jennifer (and others), and each Thurs 2pm we're having our weekly meeting on social networks and political polarization with Tian, Tom, and Julien (and others).

Brian Min sent this in:

and also this:

This story from 1952 about an author coming to Columbia with the intention to kill the editor who rejected his paper perhaps answers why peer review process involves anonymous referees.

I received an email from BMC Medical Informatics asking me to review a resubmission of an article I'd reviewed earlier. So far, no problem. I took a look at my original comments, the revised article, and the authors' responses. Still no problem, as I looked for their responses to my comments. Oh, there it is: "Re the comments made by Andrew Gelman . . ."

Hey! I thought referee reports were supposed to be anonymous! It is a good thing that I liked the paper, otherwise I would've made some quick enemies without even trying!!

Jarrett Byrnes writes,

A number of us use JAGS, so we import the results of our Bayesian analysis using coda, so I whipped up a quick mcmc2bugs script so that we could also use the functions in R2WinBUGS. I don't know if you'll find it useful (or if I've duplicated something that already exists - I don't think so) but, just in case, I've posted it here. [link fixed, thanks to lostnihilist]

Perhaps this should be combined with the as.bugs.array() function that's in R2WinBUGS.

Ubs pointed me to this entry at Mental Floss linking to this article by Graham Walker, who's described as "a co-author of the Official Rock Paper Scissors Strategy Guide (published by Simon and Schuster) and five-time organizer of the World Rock Paper Scissors Championships." Hey, I wanted to organize a RPS tournament one winter in college but everybody thought it was a silly idea. Credit goes to those who put in the effort.

Anyway, here are Walker's suggestions. I thought it was just going to be a joke--"rock always ins" and all that--but they actually look pretty good to me. The comments at the end of the article are interesting too.

The secret to winning at RPS

Basically, there are two ways to win at RPS. First is to take one throw away from your opponent options. ie - If you can get your opponent to not play rock, then you can safely go with scissors as it will win against paper and stalemate against itself. Seems impossible right? Not if you know the subtle ways you can manipulate someone. The art is to not let them know you are eliminating one of their options. The second way is to force you opponent into making a predictable move. Obviously, the key is that it has to be done without them realizing that you are manipulating them.

Most of the following techniques use variations on these basic principles. How well it works for you depends upon how well you can subtly manipulate your opponent without them figuring out what you are doing. So, now that the background is out of the way, let's get into these techniques:

1 - Rock is for Rookies

In RPS circles a common mantra is "Rock is for Rookies" because males have a tendency to lead with Rock on their opening throw. It has a lot to do with idea that Rock is perceived as "strong" and forceful", so guys tend to fall back on it. Use this knowledge to take an easy first win by playing Paper. This tactic is best done in pedestrian matches against someone who doesn't play that much and generally won't work in tournament play.

2 - Scissors on First

The second step in the 'Rock is for Rookies' line of thinking is to play scissors as your opening move against a more experienced player. Since you know they won't come out with rock (since it is too obvious), scissors is your obvious safe move to win against paper or stalemate to itself.

3 - The Double Run

When playing with someone who is not experienced at the RPS, look out for double runs or in other words, the same throw twice. When this happens you can safely eliminate that throw and guarantee yourself at worst a stalemate in the next game. So, when you see a two-Scissor run, you know their next move will be Rock or Paper, so Paper is your best move. Why does this work? People hate being predictable and the perceived hallmark of predictability is to come out with the same throw three times in row.

4 - Telegraph Your Throw

Tell your opponent what you are going to throw and then actually throw what you said. Why? As long as you are not playing someone who actually thinks you are bold enough to telegraph your throw and then actually deliver it, you can eliminate the throw that beats the throw you are telegraphing. So, if you announce rock, your opponent won't play paper which means coming out with that scissors will give you at worst a stalemate and at best the win.

5 - Step Ahead Thinking

Don't know what to do for your next throw? Try playing the throw that would have lost to your opponents last throw? Sounds weird but it works more often than not, why? Inexperienced (or flustered) players will often subconsciously deliver the throw that beat their last one. Therefore, if your opponent played paper, they will very often play Scissors, so you go Rock. This is a good tactic in a stalemate situation or when your opponent lost their last game. It is not as successful after a player has won the last game as they are generally in a more confident state of mind which causes them to be more active in choosing their next throw.

6 - Suggest A Throw

When playing against someone who asks you to remind them about the rules, take the opportunity to subtly "suggest a throw" as you explain to them by physically showing them the throw you want them to play. ie "Paper beats Rock, Rock beats scissors (show scissors), Scissors (show scissors again) beats paper." Believe it or not, when people are not paying attention their subconscious mind will often accept your "suggestion". A very similar technique is used by magicians to get someone to take a specific card from the deck.

7 - When All Else Fails Go With Paper

Haven't a clue what to throw next? Then go with Paper. Why? Statistically, in competition play, it has been observed that scissors is thrown the least often. Specifically, it gets delivered 29.6% of the time, so it slightly under-indexes against the expected average of 33.33% by 3.73%. Obviously, knowing this only gives you a slight advantage, but in a situation where you just don't know what to do, even a slight edge is better than none at all.

8 - The Rounder's Ploy

This technique falls into more of a 'cheating' category, but if you have no honour and can live with yourself the next day, you can use it to get an edge. The way it works is when you suggest a game with someone, make no mention of the number of rounds you are going to play. Play the first match and if you win, take it is as a win. If you lose, without missing a beat start playing the 'next' round on the assumption that it was a best 2 out of 3. No doubt you will hear protests from your opponent but stay firm and remind them that 'no one plays best of one for a kind of decision that you two are making'. No this devious technique won't guarantee you the win, but it will give you a chance to battle back to even and start again.

Sophia Rabe-Hesketh writes,

Mark Wilson and I are looking for two full-time postdocs for two years.

Applicants should have a Ph.D. and a strong background in quantitative methods. Some knowledge of multilevel modeling and/or item response theory would be an advantage.

The full ad is here.

I sent this Mark Buchanan article to Bruce Levin (coauthor. with Michael Finkelstein, of Statistics for Lawyers) who pointed me to this discussion from their book of what he calls the prosecutor's fallacy and the defense fallacy. Interesting stuff.

Changhe Yuan writes,

I have been looking at the Trend Analysis of the Sex Ratio at Birth in the United States. It provides a chart analogous to the one posted previously for China.

There are some paralells: in the times of war (WW2 from 1940-1945, civil war in China 1945-1953, Vietnam for US around 1970) there is a greater proportion of boys to girls. But this does not fully explain the shift towards boys in China from 1985 onwards...

Enrique Pérez Campuzano writes,