Matthieu Domenech de Cellès writes:
I am a research group leader in infectious disease epidemiology at the Max Planck Institute for Infection Biology in Berlin. I read your recent post on how to respond to fatally flawed papers.
Here is a personal anecdote from my field, which you might find interesting.
The flawed paper in question was published in the New England Journal of Medicine (NEJM), supposedly one of the most prestigious medical in the world (at least if one cares about its impact factor, which was 158.5 in 2022). The study (https://pubmed.ncbi.nlm.nih.gov/22970945/) was a case-control study aiming to estimate how the risk of pertussis varied over time after receipt of 5 doses of pertussis vaccines in children aged 4–12 in California. The main result was that, in this fully vaccinated population, the risk of pertussis increased by 42% each year after the fifth dose. The analysis itself was OK, but the fatal flaw came from the interpretation of this result, as reported in the discussion: “[…] we estimated that the fifth dose of DTaP became 42% less effective each year […]”. From this flawed interpretation, they claimed that the protection conferred by pertussis vaccines waned very fast (maybe on a time scale of a few years), hence the title. As I write this, this paper has been cited 606 times (according to Google Scholar) and has largely led the field to believe these vaccines are no good.
As the authors acknowledged, however, their study did not include an unvaccinated group, which is needed to estimate vaccine effectiveness (VE). In other words, it was impossible to conclude about the rate of waning VE from the 42% increase they estimated. Incidentally, this basic fact should have been obvious to the editor and the reviewers from the NEJM (but as you wrote in another blog post, sometimes the problem with peer review is the peers). In a follow-up study (https://pubmed.ncbi.nlm.nih.gov/31009031/), we showed that, given the high transmissibility of pertussis, this 42% was entirely consistent with vaccines with sustained effectiveness whose protection wanes, on average, slowly. More generally, we demonstrated that their claim (i.e., a 42% annual increase in pertussis risk after vaccination is equivalent to a 42% yearly decrease in VE) was baseless.
Overall, we adopted a strategy close to option 5 in your post. We initially submitted the paper to NEJM, which rejected it immediately (probably for the reasons you mentioned). Our paper was ultimately published in 2019 (in JAMA Pediatrics), with 28 citations so far. Although this was a good outcome, I don’t expect our paper ever to reach the impact of the NEJM paper, whose damage is probably irreversible.
I took a look at the JAMA Pediatrics paper, and . . . wow, this is complicated! These cohort issues really confound our intuitions. This fits into our recently-discussed “It’s About Time” theme. I feel like there should be some clever graph that could make this all clear to me. The paper does have some graphs, but not quite what it takes for me, in that I still struggle to hold this model in my head. I kinda believe that Domenech and his colleagues are correct—they sent me the email, after all!—but I don’t have the energy right now to try to untangle all this. Given that, I can see how the NEJM editors could’ve thrown up their hands here. I’d be interested in seeing their stated reasons for rejecting the Domenech et al. paper.
P.S. When I was a kid, I never heard of anyone who had whooping cough. We all wanted to get it, though, because the only thing anyone knew about whooping cough was from the school rules that they would give out every year which had a list of the minimum number of days of absence from school if you had various illnesses: measles, etc. For some of these, the minimum period of absence was 1 week or 2 weeks. The thing I remember was that whooping cough was the only illness where you’d have to stay out of school for a full 4 weeks—or maybe it was 8 weeks, I can’t recall the exact number, just that whooping cough was the longest. From a logical standpoint, this should’ve made us super-scared of whooping cough—it’s so dangerous you have to miss a month or two of school—but, to us, the lure of getting to miss 4 or 8 weeks of school was so appealing. Recess and gym class aside, elementary school was such a horrible waste of time.
Also, we had no idea whether it was supposed to be pronounced “wooping cough” or “hooping cough.”