Daniel Carlat posts a link to this news article by John Fauber about a medical researcher, James Stein, who took big bucks in lecturing and consulting fees from drug companies over a 12-year period, before stopping a few months ago. Stein said:
I was sure I could avoid bias because I controlled the content and I had these strong personal convictions. Well, unfortunately, over the past several months, I’ve learned that I was wrong. I’ve learned that I could not stay unbiased, that I could not control all the content of my talks, and that my personal convictions were not good enough.
Regarding disclosure as a potential solution, Stein said:
I really felt that if I stood up in front of a crowd and said that these are my disclosures, look how honest I am, that I was really managing conflict of interest. But actually the medical literature and the social science literature tells me that it is actually the opposite effect. Although it is laudable to disclose your relationships, actually thinking that disclosure manages relationships is harmful. It has the perverse effect that when you disclose your relationship, the recipient of your information becomes more trusting, and the social scientists also have shown us that professionals who disclose actually become more biased…. I would argue … that the solution is not disclosure, because if you are doing something that is wrong or unethical, don’t disclose, just don’t do it!
There was also this amazing bit:
Huge fines or convictions for gross ethical conduct were being issued against every drug company that he worked with. Doctors were being investigated on allegations of taking kickbacks.
Quoting Stein's epiphany "I have learned that human beings, physicians included, are incapable of recognizing bias in themselves, and even when you try not to be biased it is impossible to avoid it, especially when money is involved," when I was working in that area some years ago we were struck the fact that many who initially felt like Stein (I won't be biased) were doing clinical research where they insisted patients and outcome assessors had to blinded to treatment assignment to avoid bias – (i.e. the shoe makers children going with out shoes).
As for processing the information about potential conflicts – I would believe critical reviewers likely can get a better sense of the directions and amounts of bias – and so maybe the problem there is just poor processing of that kind of information more widely.
But in practicing statistics, we are all often in this sort of situation – of knowing that others (not all but many) will be more pleased with more striking results with less uncertainty …
Keith
p.s. "fear" of non-replication by others might help especially if its commonly insisted upon
This issue is more widespread than medical research, as I'm sure you know. As you know, most other research is funded by corporations with a stake in the outcome and even government funding has direction because it fulfills a grant request. How much does directional intent influence reporting? I'm not talking fraud or even results but the more basic fact that most studies depend on statistics and thus on how you control, how you slice the data, etc., information which very few people look at – if it's even included – and which even fewer could take apart.
We know – well, it appears, but this is a comment on a blog not academic writing – that even small gifts influence doctor recommendations and prescriptions. How much is research affected in ways that we can't adequately even see by funding, both the source of the $$ and the directional influence imposed by the grant?
My gut tells me we account poorly for these biases. We accumulate a bunch of studies – and then meta-analysis – when it seems to me the results would describe a distribution shifted from an imaginary objective center. Perhaps we should literally impose a discount factor, something I believe we do in our heads regularly.