Mark Thoma has an interesting discussion of the challenge that the economics profession, and individual economists, have when they give policy recommendations.
Mark’s basic point goes as follows. Consider the following four stages of a model:
(a) assumptions about fundamental principles of how the world works,
(b) normative principles (that is, fundamental goals, views about how the world should be),
(c) conclusions about the likely effects on policy,
(d) recommendations about policies.
In any rigorous economic model, there should be a mapping leading from (a) to (c). Further reasoning (possibly mathematical modeling, as in cost-benefit analysis) will take you from (b) and (c) to (d).
That’s all fine. But Mark’s point is that the reasoning can go the other way too: start with (b) and (d), and then you can figure out what (c) needs to be, and then you can go back one more step and figure out what model (a) you need to get started! Even if economists are not doing this reasoning-from-conclusions-to-assumptions explicitly, you could well believe it’s going on implicitly as well as being induced by various pressures such as the selection of what research results to report and even what problems to work on.
This is inevitable, and I discuss it in the decision analysis chapter (22, I think it is) of Bayesian Data Analysis. We call it the garbage-in-garbage-out problem: If you can come with any decision you’d like by just altering the inputs of your analysis, then what’s the point of decision analysis (or, by extension to the above-linked example, economic modeling) at all?
My answer is something that I call “institutional decision analysis,” which has two principles:
1. It can be a good idea to provide reasoning to justify your decisions. As an individual person, you might not have to justify your personal decisions to anyone (except to your spouse), but an institution–whether it be a business, a government agency, a nonprofit organization, or some other grouping–often needs some path of bread crumbs connecting assumptions to recommendations. (Here, I carefully say “connecting” rather than “leading from” to be agnostic about the direction of the reasoning.)
2. As Mark noted, an overall decision recommendation on anything important is likely to be so dependent on assumptions to such an extent that it’s probably fair to say that the analyst is reasoning from conclusions to assumptions (from (d) to (c) and then to (a), in my above notation). But, even then, formal decision analysis can be useful in making relative recommendations. This is the point that we made in our article about decision making for home radon [link fixed]. In the economics context, this might suggest that economists of different political persuasions could still give useful recommendations about how to spend money or cut taxes, or where in the economy such policies would make more or less sense.
Professor Gelman: This puts things very well. I like it. Thank you.
What makes research compelling, particularly in the social sciences and particularly when the positive implications speak to the crux of a big policy issue, is when the investigator bends over backwards to consider how alternative assumptions/theories are or are not disproved by the data. That is, the investigator at least attempts to be his/her own worst critic in the design of the analysis and reporting of results. When scientific peer review rewards this kind of scholarly depth over all else I believe we will see an acceleration of progress in the social sciences. This isn't always the standard in economics today.
I think your link for the radon article is off — it goes to the same post as the "institutional decision analysis" link, and there's no mention of radon there.
The narrative fallacy which is a favorite of Taleb's also is at play here. It appears that there are no patterns in economic variables that a competent economist cannot sit down and explain, typically through a long chain of if-thens, with minimal data and with maximal conviction. The absence of data in most of these exchanges is unsettling.
Starting with (b) and (d) and finding model (a) to fit the conclusions – unfortunately – is 95% of what happens in the "real world". Management consultants are trained in something called "blank sliding" which is to populate all slides with headers telling a story before gathering a shred of evidence. Investment bankers know what price they are willing to pay for an asset before they build the valuation model.
Just look at the "debates" on climate change and evolution and one begins to see a pattern replicating itself everywhere
"Can economists be trusted?"
In short: No!
If a longer justification is needed then (perhaps to to paraphrase Andrew's argument), far too little of the "dismal science" is empirical or evidence based. Far too much is motivated by ideology and dubious theoretical arguments: consumers act rationally, the market is perfectly informed & competitive etc etc.
I could go on, and on, and on…