Daniel Ho and Kevin Quinn write:

We amass a new, large-scale dataset of newspaper editorials that allows us to calculate fine-grained measures of the political positions of newspaper editorial pages. Collecting and classifying over 1500 editorials adopted by 25 major US newspapers on 495 Supreme Court cases from 1994 to 2004, we apply an item response theoretic approach to place newspaper editorial boards on a substantively meaningful–and long validate–scale of political preferences.We validate the measures, show how they can be used to shed light on the permeability of the wall between news and editorial desks, and argue that the general strategy we employ has great potential for more widespread use.

Here’s their key graph, which aligns the estimated ideological positions of major newspapers with recent Supreme Court justices:

They used Bayesian ideal point estimation. Their main substantive conclusion:

Most newspapers take political positions that are relatively centrist.We estimate that 52% of the largest 21 newspapers in our dataset take positions between the justices on either side of the median justice . . . about half of the newspapers take relatively moderate positions on issues coming before the court. Clearly, much depends on one’s prior conception of where the political center lies, but our results at least inform the relative assessment.

Finally, the results speak to the large literature on the differences between news and editorial boards. Our [Ho and Quinn’s] measures — to our knowledge the first to measure explicit editorial board positions in a systematic model-based fashion — correlate strongly with existing measures of implicit positioning, suggesting that the editorial board is likely not hermetically sealed from the news division.

Some technical comments

I like this paper a lot! But I have one small criticism, which is that I think they are overinterpreting the parameters in their models. Ho and Quinn write:

In addition, we can compare papers in a way previously impossible with any degree of precision. For instance . . . the posterior probability that the San Francisco Chronicle is more liberal than the Boston Globe is about 0.88.

I don’t think this makes sense, especially the “any degree of precision” bit. As the authors discuss later in their article, different newspapers are more or less liberal or conservative on different issues. So the parameter that represents “the liberalism of the Boston Globe” really represents the average of the newspaper’s liberalism on some number of issues. Since there are only a finite number of issues in the dataset, you can’t really estimate this average to unlimited precision (even getting beyond issues of interpretation of this average).

Also, on a technical level, I suspect they are making a mistake by discarding the first 50,000 iterations of their simulation and then keeping the next 4 million iterations. If you really need 4 million to get a good sample, than it’s hard for me to believe that only 50,000 are needed to forget the starting points. Conversely, if 50,000 is enough for burn-in, I can’t believe you need 4 million to get a good sample.

Also, on the top of page 362, couldn’t soime of these hyperparameters (for example, the location and scale of the distribution of alphas) be estimated from the data? These models have redundancy issues, so maybe I’m missing something here, but that’s what it looks like to me.

P.S. See Kevin Quinn’s reply here.