After some adjustments (patients removed, data points
guessed), a preprint was published simultaneously with a paper in a peer
reviewed journal Raoult basically controls…
It is the opposite of proper clinical research practice. The
trial started somewhere in March, likely before an ethics vote was applied for.
But already on 11 February 2020 Raoult and colleagues decreed which drug can
prevent and cure COVID-19…
As Elisabeth Bik discussed, the paper was peer-reviewed in
less than 24h, it probably helped that the journal’s Editor-in-Chief is both
the paper’s coauthor and Raoult’s IHU subordinate Jean-Marc Rolain. Bik also
listed other problems with that paper. Like this:
“In the EU Clinical Trial Register page, the study was
described as evaluating PCR data on Day 1, Day 4, Day 7 and Day 14. However,
the study show the data for Day 6, which is different than planned. Why did the
authors not show the results on Day 7? Did the data on day 7 not look as good?”…
In fact, the authors never showed the results of day 14
either. They also refused to share their secondary endpoint data, namely “the
clinical effectiveness of treatment on time to apyrexia, normalization of
respiratory rate, and average length of hospital stay and mortality”.
Basically, it is none of anyone’s business to know if the therapy had any
clinical benefit for the patients…
Even then, additional tricks were apparently needed. Viral
load in control patients was analysed by quantitative RT-PCR and a bit too
liberally, as Bik explains:
“Of particular note, control patients 6 and 8-16 appear to
have been analyzed differently. Their Day 0 PCR values are not given as CT
values (the number of cycles after which a PCR becomes positive, the lower the
number, the more virus is present) but as POS/NEG, suggesting a different test
was used. […] Several patients in the control group did not even have a PCR
result on Day 6, so it is not clear how they were counted in the Day 6 result.”..
Some control patients were tested only every second day,
afterwards guessed to be positive, some were not tested at all on day 0, as a
PubPeer commented noted. And then the authors simply changed the results in
control patients between the “in press” and the final paper version, as another
PubPeer user observed…
Another PubPeer user re-ran the analysis, because “an
important number of non-treated patients were not tested by PCR (ND)“. Once the
ND values were disregarded, there wasn’t any more significant difference
between controls and chloroquine-treated patients.
Not only control group had to be adjusted: the treatment
group somehow lost 6 patients, as Bik discussed:
“Although the study started with 26 patients in the HQ or
HQ+AZ group, data from only 20 treated patients are given, because not all
patients completed the 6-day study. The data for these 20 patients looks
incredibly nice; especially the patients who were given both medications all
recovered very fast.
What happened to the other six treated patients? Why did
they drop out of the study? Three of them were transferred to the intensive
care unit (presumably because they got sicker) and 1 died. The other two
patients were either too nauseous and stopped the medication, or left the hospital
[…] So 4 of the 26 treated patients were actually not recovering at all.”
It seems, the authors simply removed inconvenient patients
from the analysis before publishing their study. They left online their
previous evaluation though, which showed a slightly different result.
Statistical and ethical problems with that Gautret et al
2020 study were addressed extensively in this preprint by Dahly, Gates & Morris
zenodo 2020. The authors also reject Raoult’s claims about previous positive
results from China:
“In the background of their paper, Gautret et al referred to
“an early clinical trial conducted in COVID-19 Chinese patients, [which] showed
that chloroquine had a significant effect, both in terms of clinical outcome
and viral clearance, when comparing to control groups”. There were two
citations for this claim. The first was a letter7 that doesn’t report any trial
data, but instead refers to a conference held in China in February, during
which participants (“experts from government and regulatory authorities and
organizers of clinical trials”) seemingly agreed that chloroquine was an
efficacious treatment for COVID-19. The second cition (also included in the
aforementioned letter) refers to a number of clinical trials registered in
China, though many of these have now been suspended or closed, while the
remaining trials are still recruiting (per their entries on
http://www.chictr.org.cn as of March 21, 2020). Hence there are, to our
knowledge, no other published reports of clinical trials testing the efficacy
of chloroquine for COVID-19 treatment.”
BUT WHAT IF RAOULT IS RIGHT AND IT WORKS?????
I hear you ask that. After all, Raoult is a star of French
science: he publishes a scientific paper “nearly every work day”, thanks to 800
employees working under him, as a blog post mentioned, and he is so important
that he named two bacteria species after himself: Raoultella planticola and
Rickettsia raoultii.
Well, Elisabeth Bik found some very bad data fakery in a 15
year old Raoult-co-authored paper…
Indeed, problems in a paper about a mouse model for typhus
got his lab in hot water in 2006. A reviewer for Infection and Immunity, a
journal published by the American Society for Microbiology (ASM), discovered
that four figures in a revised manuscript were identical to figures in the
original manuscript, even though they were supposed to describe a different
experiment.
In letters to ASM, made available by Raoult, second author
Christian Capo and last author Jean-Louis Mège, a group leader, accepted “full
responsibility” for the problem, which they said involved only two figures.
Capo, in his letter, wrote that he had made an innocent mistake; Mège wrote
that Capo had subsequently failed to show the revised manuscript to other
authors, who were on vacation, before resubmitting it. But after consulting its
ethics panel, ASM banned all five authors, including Raoult, from publishing in
its journals for a year. “We are not entirely comfortable with the explanation
provided,” ASM officials wrote to Mège. “Misrepresentation of data … is an
affront to the ethical conduct of scientific inquiry.”
Capo and Mège accepted the decision, but Raoult wrote ASM
that he wasn’t at fault and that the “collective punishment” was “very unfair.”
He appealed the ban, also on behalf of two other co-authors, but lost. Furious,
he resigned from the editorial board of two other ASM journals, canceled his
membership in the American Academy of Microbiology, ASM’s honorific leadership
group, and banned his lab from submitting to ASM journals, in which he had
published more than 230 studies. His name has been on only two ASM journal
papers since, both published in 2010. To clear his name, Raoult sent his ASM
correspondence to French colleagues in 2007, along with a letter defending
himself. “If I had been in the United States, I would have sued,” he wrote.”…
Raoult did not like Science coverage of his genius. He made
the journal publish an Erratum, where he even wrongly accused the author of
shilling for Danone:
“IT WAS AN HONOR TO HAVE MY PROFILE PUBlished in Science
(“Sound and fury in the microbiology lab,” C. Mary, News Focus, 2 March, p.
1033). However, I was surprised that 20% of the article is devoted to the
American Society for Microbiology (ASM) story, in which I was a collateral victim
of a collective sanction (there has been no collective liability in France
since World War II). I did not manage the paper and did not even check the last
version. The mistake by C. Capo consists of a single figure inversion (not
four, as stated in the Science profile). This paper has since been published
(1). In January 2007, I was awarded one of the highest ASM honors—the ICAAC
lecture—thus clearing doubts about my scientific integrity.”
Now, the extra bit of irony is that Raoult is so full of
himself that he published in 2018 an opinion piece to teach France some…
research integrity…
Now consider this. Raoult’s past papers show falsified data,
which even resulted in his ban by ASM for one year, to which Raoult responded
with threats of lawsuit. He is a patriarchal control freak and a misogynous
bully who violently punishes all disagreement and uses threats against
whistleblowers and victims to achieve compliance. He is pathologically
resistant to criticism and believes to be infallible and omniscient:.. Raoult’s
new study on chloroquine as the cure for COVID19 is obviously flawed, at best.
Should we really trust his claims and put our all lives in
his hands?
Unfortunately, there are many potential problems with the way the data and the peer review process were handled. The discussion is still ongoing on PubPeer, with 2 posts on either the preprint (40 comments as of today) or the published version (3 comments)...
ReplyDeleteSo 4 of the 26 treated patients were actually not recovering at all. It seems a bit strange to leave these 4 patients who got worse or who died out of the study, just on the basis that they stopped taking the medication (which is pretty difficult once the patient is dead). As several people wrote sarcastically on Twitter: My results always look amazing if I leave out the patients who died, or the experiments that did not work...
The paper was submitted on 16 March and accepted, presumably after peer review, on 17 March. Assuming the paper was indeed peer reviewed within 24h, that seems incredibly fast. But there might be a reason why this process was so fast.
One of the authors on this paper, JM Rolain, is also the Editor in Chief of the journal in which the paper was accepted, i.e. the International Journal of Antimicrobial Agents. This might be perceived as a huge conflict of interest, in particular in combination with the peer review process of less than 24h.
This would be the equivalent of allowing a student to grade their own paper. Low and behold, the student got an A+!
The journal should make the peer reviews (usually reports from 2 or 3 other scientists who are not authors on the paper or work at the same institutes) publicly available, to prove that a thorough peer-review process indeed took place...
Obviously, in the case of a viral pandemic there is a great need for new results to become available as fast as possible. It is understandable that the scientific peer review process might be a bit less polished in this case than for other science papers.
However, the preprint was already publicly available for all, so it does not make sense that this peer review was done in such a rushed manner. It would have been better if the raw data were shared by preprint, with a more carefully written and analyzed, peer-reviewed version being published a couple of weeks later.
https://scienceintegritydigest.com/2020/03/24/thoughts-on-the-gautret-et-al-paper-about-hydroxychloroquine-and-azithromycin-treatment-of-covid-19-infections/
The lack of randomization limits the conclusions that should be drawn from this study.
ReplyDeleteTo conclude that the differences in the outcomes observed between the groups receiving different treatment regimens are in fact due to differences in treatment, we need to be able to confidently assert that the groups shared a similar level of baseline risk, prior to treatment.Randomization is of course our best tool for creating such comparable groups, but not applied in this study.This limitation is amplified by the fact that control patients were either recruited from other medical centres, or were patients that refused consent for the active treatment (more on this below). That said, control patients, despite experiencing worse outcomes, were younger onaverage, less likely to be male, and had a more favourable symptom profile, perhaps suggesting lower baseline risk (note: that these differences were not “significant” with a p < 0.05 does imply that they don’t matter with respect to interpretation outcomes). However, it’s entirely plausible that there could be other, unobserved factors leading to a control group with higher baseline risk. Unfortunately, there was no other information about the characteristics of patients, or about the clinicians and medical centres where they were treated, that would help us better judge the comparability of groups.
Inappropriately included control patients
Following from the points above, it’s important to note that you would not normally include someone in a trial unless they consented to receive the treatment being tested. That was not the case in this study. Instead, control patients included those who refused consent for the active treatment regimen. The fact that patients who didn’t give such consent is a red flag, not just for the interpretation of the trial but its ethics. We also think it is relevant to note that this is a treatment that the authors have been promoting in the months leading up to this study.
https://zenodo.org/record/3725560#.XoDTI4hKjIW
Catherine Mary. Sound and Fury in the Microbiology Lab. Science 02 Mar 2012:Vol. 335, Issue 6072, pp. 1033-1035. DOI: 10.1126/science.335.6072.1033
ReplyDeleteSummary
At 59, Didier Raoult is the most productive and influential microbiologist in France, leading a team of 200 scientists and students at the University of Aix-Marseille. He has discovered or co-discovered dozens of new bacteria, and in 2003, he stunned colleagues with a virus of record size, dubbed Mimivirus, the first member of a family that sheds an intriguing new light on the evolution of viruses and the tree of life. Controversial and outspoken, Raoult last year published a popular science book that flat-out declares that Darwin's theory of evolution is wrong. And he was temporarily banned from publishing in a dozen leading microbiology journals in 2006. Scientists at Raoult's lab say they wouldn't want to work anywhere else. Yet Raoult is also known for his enmities and his disdain for those who disagree with him.