Someone who wishes to remain anonymous writes:
I was consulting with an academic group, and I noticed and pointed out what I believed was a clear and obvious mistake in something they were planing to publish on. Now we can always be wrong, but here the mistake was mathematical and had been recently published by a well know author – not me. So I am very sure about it and after the consulting came to an end I sent a very blunt email along with a warning that they would not want their error to be pointed in a letter to the editor. They were arguing that the error was subtle and maybe not a really an error or at least one that was well or widely understood. (If they accept it is an error they would have to develop a new method.)
Unfortunately/fortunately I noticed a recent paper from them was listed in the references of a statistical methods paper I was just asked to review. I checked the paper and the error is there with not even a warning about any uncertainty or mixed opinions about there possibly being an error. There was no signed non-disclosure clause. What should I do now?
I can’t find someone else to write the letter to the editor I will but I am wondering how often others run into this and what ideas they have.
P.S. To preserve anonymity, some of the details have been faslified in unimportant ways.
My reply:
Many years ago I was involved in a project–not as a coauthor but just as a very peripheral member of a research group–where the data collection didn’t go so well, the experimenters got a lot fewer participants than they had hoped for, and when all was said and done, there didn’t seem to be much difference between treated and control groups. We were all surprised–the treatment made a lot of sense and we all thought it would work. After it didn’t, we had lots of theories as to why that could’ve been, but we didn’t really know. One other thing: there were before and after measures (of course), and both the treatment and the control groups showed strong (and statistically significant) improvement.
I drifted away from this project, but later I heard that the leader of the study had published a paper just with the treatment group data. Now, if all you had were the treatment data–if it were a simple before/after study–that would be fine: there would be questions about causality but you go with what you’ve got.
I didn’t follow up on this, partly because I don’t know the full story here. The last I saw of the data, there wasn’t much going on, but it may very well be that completely proper data analysis led to cleaner results which were ultimately publishable. To say it again for emphasis: I don’t really know what was happening here; I only heard some things second-hand. In any collaboration, it’s good to have a certain level of communication, trust, and sense of common purpose which just wasn’t there in that project at all.
Anyway, back to the original question: I don’t see why you can’t yourself write and submit a letter to the editor. First run it by the authors of the article to see what they say, then send it to the journal. That would seem like the best option for all concerned. Ideally it won’t be perceived as a moral issue but just as a correction. As the author of a published paper with a false theorem, I’m all in favor of corrections!
Editors are not always willing to publish corrections. In that case you could go to a forum such as BioMed Critical Commentary to discuss errors.
Or, for that matter, I could publish the correction on this blog.
If you do try writing a letter to the journal I would be interested to find out what happens.
I have some experience with this sort of thing which has been uniformly negative. The most recent was discussed on this blog just a couple of weeks ago under the heading "Conflict over Conflict-Resolution Research" (for some reason I'm having trouble pulling the link off of this web sit). One aspect of this is that there are a number of straight errors in the piece that we critique. This was published in the British Medical Journal but the BMJ refuses to make corrections. By the way, some of these errors are exposed in the appendix to our article which, understandably, not a lot of people have read:
http://www.hsrgroup.org/images/stories/Documents/…
In a second experience, a pretty well-known journal refused to correct a number of errors in an article they published. We wrote a letter correcting some of the more important errors in the paper but they said that they do not publish letters. We replied that publishing a letter isn't important to us as long as they correct the errors but the editors have refused. We wrote several times asking if the journal has any policy correcting errors but they never responded to this question.
I had similar experiences with a newspaper and a magazine but with a twist. In both cases I presented editors with a list of errors and in both cases they published a correction on just one of the errors.
Readers may recall that Andrew commented on this fabulous paper:
http://www.stat.columbia.edu/~cook/movabletype/ar…
The was also quite a good article in (ironically) the BMJ on "citation errors" which gives one angle on how falsehoods enter and entrench themselves in the scietific literature:
http://www.bmj.com/cgi/content/abstract/339/jul20…
Mike Spagat
I don't have that much experience, but I'd say don't be shy about writing a letter to the editor. If they won't publish it you can always write it up as a short paper for a different journal. It basically gives you an easy publication, and it will be cited by other people who want to point out the same mistake in their work.
I see ways this can go south.
From the authors' point of view, you told them, they rejected your arguments and published.
You were not important enough to the research to be a co-author or footnote.
If this is an applied environment — you were hired, you did your job, you expressed your concerns, and you are expected not to bite the hand that fed you.
If this is an academic environment, there mayb e some relationship between you area (say, stats) and their area (say, economics) that may become frayed if there's a public tiff. From the department chairman's point of view, the relationship with econ may be more important than the relationship with you.
In short, you should be prepared for unpleasantness.
I can't gauge whether it's worth it to either you or science. If people are going to die because of an inappropriate treatment, that's one thing. If people are going to think that beer advertising works more than it really does, that's another.
Is it possible to put your concerns about this older paper into the review you are doing of the newer paper? In that way, you can get the current authors to do some of the heavy lifting.
I've come across data I thought were fraudulent (made-up) three times. First time I wrote to the editor. Nothing happened. The editor didn't even reply, if I remember correctly. The second time I wrote a paper about it. Paper published. Perfectly good paper. The third time a colleague and I wrote several papers about it (the fraud appeared to involve several papers). Papers published. Led to a good relationship with the editor of the journal where the offending paper appeared. Each time my reasoning was the same: This stuff matters. If I fail to try to improve things in a case where I actually can, why bother trying in all those cases where I have almost no leverage, such as giving money to charity? The lesson I learned from my experiences is don't bother complaining to an editor, just write a paper about it.
Andrew, I'm puzzled you don't discuss the case where you did do something — the sex-ratio stuff.
Seth, one downside I can think of with writing a paper (other than possible extra work) is that if it is a methodology paper, the letter to the editor _may_ be noticed by more (from many feilds who migh use the paper) than even a paper in the same journal.
This has happened to me when I chose not to write a letter to the editor (because I worked with the authors and one was my boss) but instead published the correction in another paper – or at least their paper has had 1000,s of citations and the one with the correction maybe 10 (and in a different journal)?
But writing a paper likley would be the best route (especially given better literature searching) and I like your argument about stuff mattering and trying when you have high leverage opportunities …
Here though they seem to have _failed_ as a consultant to persuade the authors when the error could have been _contained_ and maybe its even more important for them to try.
Keith