Advertisement

Rational for meta-analysis and randomized treatment: the COVID-19 example

Published:October 21, 2020DOI:https://doi.org/10.1016/j.cmi.2020.10.012
      My opinion is that hydroxychloroquine has become the symbol of a struggle between practising physicians and methodologists [
      • Paul M.
      Has the door closed on hydroxychloroquine for SARS-COV-2?.
      ], and the Western world against the rest of the world [
      • Million M.
      • Roussel Y.
      • Raoult D.
      Chloroquine and COVID-19: a western medical and scientific drift?.
      ]. This leads to great confusion in the literature between, on the one hand, the advocates of an empirical approach based on the sensitivity of bacteria, viruses or parasites to anti-infectious agents in vitro and the rational use of these anti-infectious agents in patients, and, on the other hand, the analysts who, taking up the various studies, are more specifically interested in the form of the studies to determine the existence of biases. Recently, hydroxychloroquine, from my point of view, became a paradigm of such conflict.
      For example, when testing hydroxychloroquine, exclusion of patients without confirmed diagnosis is for us a major issue. In contrast, several studies do not consider this parameter to be essential [
      • Recovery
      Randomised evaluation of COVID-19 therapy (Recovery).
      ,
      • Skipper C.P.
      • Pastick K.A.
      • Engen N.W.
      • Bangdiwala A.S.
      • Abassi M.
      • Lofgren S.M.
      • et al.
      Hydroxychloroquine in nonhospitalized adults with early COVID-19: a randomized trial.
      ], including a large study testing the efficacy of hydroxychloroquine prophylaxis [
      • Boulware D.R.
      • Pullen M.F.
      • Bangdiwala A.S.
      • Pastick K.A.
      • Lofgren S.M.
      • Okafor E.C.
      • et al.
      A randomized trial of hydroxychloroquine as postexposure prophylaxis for covid-19.
      ]. This inclusion criterion is mandatory for me. In addition, there are considerable differences in dosing regimens as there is no standard dosing regimen: the Recovery Trial gave a theoretically toxic dosage at baseline (2.4 g), others use 200 mg daily, while we prescribe, in my institute, 600 mg daily, as in Q fever or Whipple's disease [
      • Gautret P.
      • Million M.
      • Kaplanski G.
      • Camoin-Jau L.
      • Colson P.
      • Fenollar F.
      • et al.
      Natural history of COVID-19 and therapeutic options.
      ]. Also, it is important to compare the duration of treatment. Entering hospital data with a hydroxychloroquine yes/no answer does not tell much about the treatment. These are major problems with the inclusion criteria in big data studies.
      The stage of the disease at which treatment is given is critical. Two reviews explain that there are four stages in the disease [
      • Gautret P.
      • Million M.
      • Kaplanski G.
      • Camoin-Jau L.
      • Colson P.
      • Fenollar F.
      • et al.
      Natural history of COVID-19 and therapeutic options.
      ,
      • McCullough P.A.
      • Kelly R.J.
      • Ruocco G.
      • Lerma E.
      • Tumlin J.
      • Wheelan K.R.
      • et al.
      Pathophysiological basis and rationale for early outpatient treatment of SARS-CoV-2 (COVID-19) infection.
      ]. There is a first virological stage, when the antiviral drugs can be effective, a second viro-immunological stage, when the immunological reaction aggravates the patient's condition and is associated with abnormal coagulation phenomena including anti-phospholipid antibodies [
      • Bertin D.
      • Brodovitch A.
      • Beziane A.
      • Hug S.
      • Bouamri A.
      • Mege J.L.
      • et al.
      Anti-cardiolipin IgG autoantibodies are an independent risk factor of COVID-19 severity.
      ], a third stage that is exclusively or almost exclusively immune, also called the cytokine storm, and finally a fourth stage, resulting from multiple pulmonary and visceral injuries. Thus, each stage probably corresponds to different therapeutic strategies. Finally, evaluation of the therapeutic efficiency in all infections of the lung is usually performed after 3 days of treatment. Mixing patients of different stages, with different doses and durations of treatment, may result in a false result known as “Simpson's paradox” [
      • Janket S.J.
      • Ackerson L.K.
      • Diamandis E.
      Simpson’s paradox in proof-of-concept studies.
      ]: studying separate groups of patients from the same study may result in opposite conclusions. Some French studies exclude patients who are treated while they have already been hospitalized for 2 days [
      • Mahévas M.
      • Tran V.T.
      • Roumier M.
      • Chabrol A.
      • Paule R.
      • Guillaud C.
      • et al.
      Clinical efficacy of hydroxychloroquine in patients with covid-19 pneumonia who require oxygen: observational comparative study using routine care data.
      ], which is difficult to understand. For sure, a key element is the delay between the start of disease and implementation of the treatment.
      Finally, we need outcomes that are measurable and not too dependent on circumstances. Transfer to the critical care unit is dependent on local management and possibilities. Hospitalization is not a good criterion either. The decision to hospitalize is not based on objective criteria, but rather reflects a management strategy or bed availability rather than a clear clinical reality. These are reflections that have arisen from our experience here having followed more than 6000 people with this disease [
      • Lagier J.C.
      • Million M.
      • Gautret P.
      • Cortaredona S.
      • Giraud-Gatineau A.
      Outcomes of 3,737 COVID-19 patients treated with hydroxychloroquine/azithromycin and other regimens in Marseille, France: a retrospective analysis.
      ]. Evaluation of outcome, as in other infectious diseases, can be clinical with objective measures (death) or biological. In fact, viral loads allow better appreciation of the effectiveness of an antiviral treatment (especially in AIDS and hepatitis C) than the design of the study. Indeed, the incontestable objective factors are death and viral load, which are measurable elements and which do not (or only slightly for PCR) depend on the observer or the circumstances. Of course, they may not be linked directly, but both report on some objective efficacy of the treatment.
      Thus, if we look at the practice of care using two therapeutic reference books as a reference, Conn's current therapy book [
      • Kellerman R.
      Conn’s current therapy.
      ] and the Bennet Dolin Blaser book [
      • Bennett J.E.
      • Dolin R.
      • Blaser M.J.
      Mandell, Douglas, and Bennett’s principles and practice of infectious diseases.
      ], there are very few cases of treatable infections where randomized studies have been able to modify or allow the management of infectious diseases. In bacteriology, antibiotic testing drives therapeutics. In virology, viral load drives therapeutics, as for AIDS or hepatitis C. In parasitology, for therapeutics, effects (for malaria) have also been biologically measured. Thus, in contrast to the asserted dogma, there are very few examples of currently followed therapeutics in infectious diseases that have been determined by randomized double-blind studies [
      • Anglemyer A.
      • Horvath H.T.
      • Bero L.
      Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials.
      ]. Methods of randomized trials have dogmatically become the reference standard, but are controversial [
      • Woodcock J.
      • Ware J.H.
      • Miller P.W.
      • McMurray J.J.V.
      • Harrington D.P.
      • Drazen J.M.
      Clinical trial series.
      ,
      • Frieden T.R.
      Evidence for health decision making – beyond randomized, controlled trials.
      ,
      • Bédécarrats F.
      • Guérin I.
      • Roubaud F.
      Randomized control trials in the field of development – a critical perspective.
      ]. Currently randomized trials have been severely criticized including in cardiology and cancer therapy [
      • Kuss O.
      • Legler T.
      • Börgermann J.
      Treatments effects from randomized trials and propensity score analyses were similar in similar populations in an example from cardiac surgery.
      ,
      • Edwards J.P.
      • Kelly E.J.
      • Lin Y.
      • Lenders T.
      • Ghali W.A.
      • Graham A.J.
      Meta-analytic comparison of randomized and nonrandomized studies of breast cancer surgery.
      ]. A recent giant review by the Cochrane Library (including 1583 meta-analysis covering 228 medical conditions) fails to show any superiority of randomized studies versus observational studies in many healthcare outcomes [
      • Anglemyer A.
      • Horvath H.T.
      • Bero L.
      Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials.
      ]. It is not established therefore that there is evidence of a superiority of randomized studies. The very existence of meta-analysis highlights that there are discrepancies between the different randomized studies, which proves that these studies did not eliminate biases [
      • Lagier J.C.
      • Million M.
      • Raoult D.
      Bouillabaisse or fish soup: the limitations of meta-analysis confronted to the inconsistency of fecal microbiota transplantation studies.
      ]. Indeed in this journal, as for hydroxychloroquine, it was reported that one meta-analysis demonstrates no effect of hydroxychloroquine on COVID-19 infection [
      • Fiolet T.
      • Guihur A.
      • Rebeaud M.
      • Mulot M.
      • Peiffer-Smadja N.
      • Mahamat-Saleh Y.
      Effect of hydroxychloroquine with or without azithromycin on the mortality of COVID-19 patients: a systematic review and meta-analysis.
      ] and the other the opposite [
      • Million M.
      • Gautret P.
      • Colson P.
      • Roussel Y.
      • Dubourg G.
      • Chabrière E.
      • et al.
      Clinical efficacy of chloroquine derivatives in COVID-19 infection: comparative meta-analysis between the big data and the real world.
      ]. However, meta-analysis exemplified the role of the sponsor in the biases, as shown by us for probiotics [
      • Million M.
      • Raoult D.
      Publication biases in probiotics.
      ].
      To evaluate observational studies, after analysing the significance of each factor, including therapy, on the outcome, it is possible to carry out multivariate analysis that tests the independence of the factors or to carry out propensity scores that should in principle neutralize a number of factors [
      • Lagier J.C.
      • Million M.
      • Gautret P.
      • Cortaredona S.
      • Giraud-Gatineau A.
      Outcomes of 3,737 COVID-19 patients treated with hydroxychloroquine/azithromycin and other regimens in Marseille, France: a retrospective analysis.
      ], or at least those that are known. Indeed, it has been well accepted for a very long time, within the framework of Simpson's paradox [
      • Stang A.
      • Poole C.
      • Kuss O.
      The ongoing tyranny of statistical significance testing in biomedical research.
      ], that not all equivocal factors are necessarily recognized, and that sometimes the addition of studies that are all in favour of one therapy gives an inverse result when combined, due to a bias in the number of people included. This paradox is well known and shows that it is impossible to fight bias mathematically and that it is necessary to have knowledge of the disease and formulate a hypothesis before interpreting massive studies [
      • Lagier J.C.
      • Million M.
      • Gautret P.
      • Cortaredona S.
      • Giraud-Gatineau A.
      Outcomes of 3,737 COVID-19 patients treated with hydroxychloroquine/azithromycin and other regimens in Marseille, France: a retrospective analysis.
      ].
      All in all, there is no indisputable science of therapeutic trials and their evaluation. It cannot be said that significant progress has been made in the practice of care by randomized trials in infectious diseases. They have generated a new specialty, particularly in the medical world, which is that of methodologists and analysts who, by definition, are convinced that their method is the best. In principle, over the history of hydroxychloroquine, depending on the studies that one decides to exclude, one is likely to retain one hypothesis or another.
      In conclusion, there are currently nearly 100 publications available in the literature evaluating, through randomized or observational studies, or big data analyses, the effect of hydroxychloroquine on patients generating opposite results. In order to be useful, a study, whatever the mode used, must first contain only patients whose diagnosis has been formally confirmed. Secondly, the stage of the disease must be specified. The effect of drugs, in most viral diseases (such as zoster, for example), is different: at the beginning, where antivirals are effective; at the time of the inflammatory reaction, where corticosteroid therapy is effective; and at the time of necrotic lesions, where no treatment is effective.
      Thirdly, the dosage of the drugs used must be clearly stated, as well as the duration of treatment.
      Fourth, anti-infective treatment cannot be evaluated unless there has been an opportunity to give it for 3 days or more for drugs with a relatively short half-life.
      Fifth, treatment can only be compared with data that are the same, whatever the conditions and circumstances, and that do not lead to subjective or ad hoc evaluations; then, the primary objectives must be death and disappearance of the virus.
      Sixth, the patients should be compared in the high-risk groups, including age, lymphopenia and level of oxygen dependence.
      These elements constitute the basis, whatever the method of inclusion chosen (randomized or not) or the method of analysis. This represents, in my opinion, the medical basis of therapeutic evaluation. Moreover, the data processing and the redaction of the paper, in my opinion, must absolutely be carried out by neutral structures, having no conflict of interest, and not by the pharmaceutical industry and be available, at the latest, 1 year after the publication of the work in order to be evaluated by the other teams.

      Transparency declaration

      The author declares no conflict of interest.

      Funding source

      No funding.

      References

        • Paul M.
        Has the door closed on hydroxychloroquine for SARS-COV-2?.
        Clin Microbiol Infect. 2020; https://doi.org/10.1016/j.cmi.2020.10.011
        • Million M.
        • Roussel Y.
        • Raoult D.
        Chloroquine and COVID-19: a western medical and scientific drift?.
        Eur J Intern Med. 2020; 78: 4-5
        • Recovery
        Randomised evaluation of COVID-19 therapy (Recovery).
        University of Oxford, 2020 (Available at:)
        • Skipper C.P.
        • Pastick K.A.
        • Engen N.W.
        • Bangdiwala A.S.
        • Abassi M.
        • Lofgren S.M.
        • et al.
        Hydroxychloroquine in nonhospitalized adults with early COVID-19: a randomized trial.
        Ann Intern Med. 2020; 173: 623-631
        • Boulware D.R.
        • Pullen M.F.
        • Bangdiwala A.S.
        • Pastick K.A.
        • Lofgren S.M.
        • Okafor E.C.
        • et al.
        A randomized trial of hydroxychloroquine as postexposure prophylaxis for covid-19.
        N Engl J Med. 2020; 383: 517-525
        • Gautret P.
        • Million M.
        • Kaplanski G.
        • Camoin-Jau L.
        • Colson P.
        • Fenollar F.
        • et al.
        Natural history of COVID-19 and therapeutic options.
        Exp Rev Clin Immunol. 2020; (In press)
        • McCullough P.A.
        • Kelly R.J.
        • Ruocco G.
        • Lerma E.
        • Tumlin J.
        • Wheelan K.R.
        • et al.
        Pathophysiological basis and rationale for early outpatient treatment of SARS-CoV-2 (COVID-19) infection.
        Am J Med. 2020; 7 (S0002-9343(20)30673-2)https://doi.org/10.1016/j.amjmed.2020.07.003
        • Bertin D.
        • Brodovitch A.
        • Beziane A.
        • Hug S.
        • Bouamri A.
        • Mege J.L.
        • et al.
        Anti-cardiolipin IgG autoantibodies are an independent risk factor of COVID-19 severity.
        Arthritis Rheumatol. 2020; https://doi.org/10.1002/art.41409
        • Janket S.J.
        • Ackerson L.K.
        • Diamandis E.
        Simpson’s paradox in proof-of-concept studies.
        Nat Med. 2019; 25: 1640
        • Mahévas M.
        • Tran V.T.
        • Roumier M.
        • Chabrol A.
        • Paule R.
        • Guillaud C.
        • et al.
        Clinical efficacy of hydroxychloroquine in patients with covid-19 pneumonia who require oxygen: observational comparative study using routine care data.
        BMJ. 2020; 369: m1844
        • Lagier J.C.
        • Million M.
        • Gautret P.
        • Cortaredona S.
        • Giraud-Gatineau A.
        Outcomes of 3,737 COVID-19 patients treated with hydroxychloroquine/azithromycin and other regimens in Marseille, France: a retrospective analysis.
        Travel Med Infect Dis. 2020; 36: 101791
        • Kellerman R.
        Conn’s current therapy.
        1st ed. Elsevier, Philadelphia2019
        • Bennett J.E.
        • Dolin R.
        • Blaser M.J.
        Mandell, Douglas, and Bennett’s principles and practice of infectious diseases.
        8th ed. Elsevier, Philadelphia2014
        • Anglemyer A.
        • Horvath H.T.
        • Bero L.
        Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials.
        Cochrane Syst Rev. 2014; 4MR000034
        • Woodcock J.
        • Ware J.H.
        • Miller P.W.
        • McMurray J.J.V.
        • Harrington D.P.
        • Drazen J.M.
        Clinical trial series.
        N Engl J Med. 2016; : 2167
        • Frieden T.R.
        Evidence for health decision making – beyond randomized, controlled trials.
        N Engl J Med. 2017; 377: 465-475
        • Bédécarrats F.
        • Guérin I.
        • Roubaud F.
        Randomized control trials in the field of development – a critical perspective.
        Oxford University Press, London2020
        • Kuss O.
        • Legler T.
        • Börgermann J.
        Treatments effects from randomized trials and propensity score analyses were similar in similar populations in an example from cardiac surgery.
        J Clin Epidemiol. 2011; 64: 1076-1084
        • Edwards J.P.
        • Kelly E.J.
        • Lin Y.
        • Lenders T.
        • Ghali W.A.
        • Graham A.J.
        Meta-analytic comparison of randomized and nonrandomized studies of breast cancer surgery.
        Can J Surg. 2012; 55: 155-162
        • Lagier J.C.
        • Million M.
        • Raoult D.
        Bouillabaisse or fish soup: the limitations of meta-analysis confronted to the inconsistency of fecal microbiota transplantation studies.
        Clin Infect Dis. 2020; 70: 2454
        • Fiolet T.
        • Guihur A.
        • Rebeaud M.
        • Mulot M.
        • Peiffer-Smadja N.
        • Mahamat-Saleh Y.
        Effect of hydroxychloroquine with or without azithromycin on the mortality of COVID-19 patients: a systematic review and meta-analysis.
        Clin Microbiol Infect. 2020; 27: 19-27
        • Million M.
        • Gautret P.
        • Colson P.
        • Roussel Y.
        • Dubourg G.
        • Chabrière E.
        • et al.
        Clinical efficacy of chloroquine derivatives in COVID-19 infection: comparative meta-analysis between the big data and the real world.
        New Microbe. New Infect. 2020; 38 (eCollection 2020 Nov): 100709https://doi.org/10.1016/j.nmni.2020.100709
        • Million M.
        • Raoult D.
        Publication biases in probiotics.
        Eur J Epidemiol. 2012; 27: 885-886
        • Stang A.
        • Poole C.
        • Kuss O.
        The ongoing tyranny of statistical significance testing in biomedical research.
        Eur J Epidemiol. 2010; 25: 225-230

      Linked Article