Statistical Modeling, Causal Inference, and Social Science: Is this blog too critical of innovative research?

Is this blog too critical of innovative research?

Andrew Oswald (author of the paper that found that parents of daughters are further to the left, politically, than parents of sons) writes,

I read your post on Kanazawa. I don't know whether his paper is correct, but I wanted to say something slightly different. Here is my concern.
The whole spirit of your blog would have led, in my view, to a rejection of the early papers arguing that smoking causes cancer (because, your eloquent blog might have written around 1953 or whenever it was exactly, smoking is endogenous). That worries me. It would have led to many extra people dying.

I can tell that you are a highly experienced researcher and intellectually brilliant chap but the slightly negative tone of your blog has a danger -- if I may have the temerity to say so. Your younger readers are constantly getting the subtle message: A POTENTIAL METHODOLOGICAL FLAW IN A PAPER MEANS ITS CONCLUSIONS ARE WRONG. Such a sentence is, as I am sure you would say, quite wrong. And one could then talk about type one and two errors, and I am sure you do in class.

Your blog is great. But I often think this.

I appreciate it is a fine distinction.

In economics, rightly or wrongly, referees are obsessed with thinking of some potential flaw in a paper. I teach my students that those obsessive referees would have, years ago, condemned many hundreds of thousands of smokers to death.

I replied as follows:

Jonathan

Once again, the answer to this question is found in Abelson's Statistics as Principled Argument, particular chapters 8 and 9. Amusingly, he uses the methodological problems with lung cancer causation as an example. Andrew's instincts are right. When you already believe the argument, methodological flaws can be assumed away. When the result is not credible (or at least highly surprising), it's more likely to be a result of methodological problems than anything else... this is a simple Bayesian intuition, since the vast majority of highly surprising results are wrong. Methodological flaws don't make the underlying conclusion wrong, but they meake it easy to reject a probably wrong result. Abelson (p. 171) "Characteristically, the critic who disbelieves a research claim primarily for conceptual reasons will nevertheless bolster his case by putting forward one or more methodological objections, It is contrary to scientific norms to reject an empirical finding solely because the critic does not believe it." He then goes on to discuss what happens to a claim in the presence of methodological fishiness.

brent

Oswald: "It would have led to many extra people dying."

For what it's worth, your writing in no way leads me to suspect that your hypothetical 1950's preincarnation would be professionally reckless to the point of manslaughter. I might rethink that if you ever critique my methods, of course. But right now you seem a right fine chap.