Skepticism about empirical studies?

Nick Firoozye writes,

I [Firoozye] wanted to point your attention to the following podcast by Ian Ayres on Supercrunchers, where he shows himself an enthusiastic (if perhaps a bit naïve) proponent of the statistical method. Entertaining, definitely. One thing though that I thought you might be interested in is Russ Roberts’ (the interviewer’s) own skepticism over the econometric method, which I think probably warrants a response. It may be that Roberts’ own view is due to his now-Austrian economics slant (i.e., somewhat anti-formallist approach) or perhaps to the fact that mainstream econometrics is a frequentist pursuit and one might question the honesty of the results as a consequence.

I don’t really have much to add here, except that the problem noted by Roberts (it’s hard to know whether to believe a statistical study) is even more of a problem with non-statstical empirical studies (i.e., anecdotes). I think Roberts might be overstating the problem because he is focusing on issues where he already had a strong personal opinion even before seeing data analyses. (He mentions the examples of concealed handguns and anti-theft devices on cars.) But there are a lot of areas where we have only weak opinions which can indeed be swayed by data (see here for some examples). These cases are important in their own right and also can serve as benchmarks for the success of statistical analysis, so that we can trust good analyses more when they’re applied to tougher problems. This is one way that applied statistics proceeds, by exemplary analyses of problems that might not be hugely important on their own terms but serve as useful templates. Consider, for example, the book by Snedecor and Cochran: it’s full of examples on agricultural field trials. Sure, these are important, but these methods have been useful in so many other fields. This is a great example, actually: Snedecor and his colleagues worked on agricultural trials because they cared about the results–these were not “toy examples” or thought experiments–and the resulting methods endured.

1 thought on “Skepticism about empirical studies?

  1. Part of Roberts skepticism seems to be due to his focus on causal effects. Strictly speaking causality just cannot be studied through empirical means as it is the result of a thought experiment that cannot happen. Something that cannot happen also cannot be observed, which is a problem when you want to do empirical research. Tos use Roberts example: The causal effect he seems to be after is: what would the crime rate have been if we could go back in time and disallow wearing concealed hand guns.

    Any attempt to empirically study causal effects will in one form or another contain some sort of extrapolation towards the treatment that did not happen. Often this is fine, this means that nothing unexpected is happening when applying the treatment. However, equaly often unexpected things do happen (the future is never what it used to be), and more importantly extrapolations are very sensitive to model choice. Neither of these objections can be solved through empirical means as that would require us knowing what would happen in the counterfactual case.

Comments are closed.