Carrie noticed an article in the Carlat Report describing some methods used in sponsored research to induce bias in drug trials:
1. Make sure your drug has a dosage advantage. This way, you can present your findings as a “head-to-head” trial without worrying that your drug will be outperformed. Thus, a recent article on Cymbalta concluded that “in three comparisons, the mean improvement for duloxetine was significantly greater than paroxetine or fluoxetine.” (Depression and Anxiety 2003, 18; 53-61). Not a surprising outcome, considering that Cymbalta was ramped up to a robust 120 mg QD, while both Prozac and Paxil were kept at a meek 20 mg QD.
2. Dose their drug to cause side effects. . . . The original Lexapro marketing relied heavily on a study comparing Lexapro 10 mg and 20 mg QD with Celexa 40 mg QD—yes, patients in the Celexa arm were started on 40 mg QD (J Clin Psychiatry 2002; 63:331-336). The inevitably higher rate of discontinuation with high-dose Celexa armed Forest reps with the spin that Lexapro is the best tolerated of the SSRIs. . . .
3. Pick and choose your outcomes. If the results of the study don’t quite match your high hopes for the drug, start digging around in the data, and chances are you’ll find something to make you smile! Neurontin (gabapentin) is a case in point. . . .
4. Practice “creative writing” in the abstract.
Carlat also cites a study from the British Medical Journal finding that “Studies sponsored by pharmaceutical firms were four times more likely to show results favoring the drug being tested than studies funded by other sources.”
I don’t know enough about medical trials to have a sense of how big a problem this is (or, for that matter, how to compare the negatives of biased research to the positives associated with research sponsorship), but at the very least it would seem to be a great example for that “how to lie with statistics” lecture in an intro statistics class.
One thing that interests me about Carlat’s methods is that only item 3 (“Pick and choose your outcome”) and possibly item 4 (“Practice creative writing”) fit into the usual “how to lie with statistics” framework. Items 1 and 2, which involve rigging the design, are new to me. So maybe this would be a good article for an experimental design class.
For more examples and discussion, see the article by Daniel Safer in Journal of Nervous and Mental Disease 190, 583-592 (2002), cited by Carlet.
A tactic similar to (#3) is apparently widely used by tobacco companies. The company statisticans surf through large databases of observational studies, looking for small subgroups/outcomes where the results will fit their requirements. Then they arrange for research funding to be given an academic to look at that particular subgroup/outcome and 'independently' find and publish the result that nobody would believe if it came directly from the company.
I should think that there would be difficulty in using both 1. and 2. in the same study.
Wouldn't they base the commercial dosage on the dosage in the study? If so, then 1. would probably cost more and 2. would probably be less effective…
All this is now available in a professional package:
David L Sackett, Andrew D Oxman (2003). HARLOT plc: an amalgamation of the world's two oldest professions. BMJ 327: 1442-1445.
Unfortunately, method 1 costs "Ocean-front property in New Jersey", method 3 is worth nx10^3 shares of stock, and 4 is somewhere between Diamonds and 0.5% of gross sales.
Bob
Something puzzles me. How are test dosages developed? I work in a somewhat different field where no assumption is made a priori as to the magnitude of the effect of a given "treatment" or variable. Rather, the magnitude is determined via logistic regressions on a mass of heterogeous data, with a coarse-classification stage to try and ID non-linear effects. Since #1 is being presented as a way to "fix" your study, I assume that there's some other accepted methodology besides using a standard dosage for a previously existing drug?
Some drugs have a range of recommended doses, with no starting dose, so a doctor is free to choose. Interestingly the prescriber information for Cymbalta says that it is no more effective at 120 than 60 mg/day. Dose for Prozac is at the lowest recommended dose.