A living systematic review and network meta-analysis published in the British Medical Journal has analysed drug treatments used for COVID-19 (Siemieniuk et al, 2020). The objective was to compare the effects of these treatments, using data sources such as the World Health Organization (WHO) COVID-19 database, which contains a comprehensive multilingual source of global COVID-19 literature, including dates up to 3 December 2020, alongside six Chinese databases up to 12 November 2020.
On the study
The study involved a selection of randomised clinical trials in which people with suspected, probable or confirmed COVID-19 were allocated at random to drug treatment, standard care, or placebo. Pairs of reviewers then independently screened potentially eligible articles. Following duplicate data abstraction, a Bayesian network meta-analysis was conducted. Risk of bias of the included studies was assessed using a modification of the Cochrane risk of bias 2.0 tool, and the certainty of the evidence using the grading of recommendations assessment, development and evaluation (GRADE) approach. The team classified interventions for each outcome into groups from the most to the least beneficial or harmful, following GRADE guidance (Siemieniuk et al, 2020).
A total of 85 trials with 41 669 patients met the inclusion criteria as of 21 October 2020. Fifty (58.8%) trials and 25 081 (60.2%) patients were newly selected since the last iteration of this analysis; 43 (50.6%) trials that evaluated treatments with at least 100 patients or 20 events met the threshold for inclusion in the analyses and can each be sourced from the original study. Compared with standard care, corticosteroids were found to probably reduce death, mechanical ventilation, and days free from mechanical ventilation. The researchers found that the impact of remdesivir on mortality, mechanical ventilation, length of hospital stay, and duration of symptoms is uncertain, but they agreed it is unlikely to substantially increase adverse effects leading to drug discontinuation. Azithromycin, hydroxychloroquine, lopinavir/ritonavir, interferon-beta, and tocilizumab were found to probably not reduce risk of death or have an effect on any other patient-important outcome. The certainty in effects for all other interventions was found to be low or very low (Siemieniuk et al, 2020).
On the evidence
According to the researchers, this living systematic review and network meta-analysis presents a comprehensive overview of the evidence related to drug treatments for COVID-19 up to 21 October 2020 and a comprehensive list of drug trials up until 3 December 2020. They found that the certainty of the evidence for most of the comparisons was very low. Siemieniuk et al (2020) discussed that corticosteroids probably reduce the risk of death and mechanical ventilation, and increase ventilator-free days, and that while remdesivir's effects remain uncertain for this virus, they state that if one is to believe the subgroup effect, remdesivir may reduce or have no effect on mortality in patients with non-critical disease and may increase or have no effect on mortality in patients with critical illness. However, the authors note that this subgroup effect is only moderately credible, and whether remdesivir reduces or increases mortality in any subgroup is uncertain.
Siemieniuk et al (2020) point out that direct evidence from randomised controlled trials in patients with COVID-19 has thus far provided little definitive evidence about adverse effects for most interventions, apart from remdesivir, which likely has low risk for adverse effects leading to drug discontinuation. They highlight that no other drug treatment was found to be impactful on any patient with at least moderate certainty for any other outcome, and that based on three small trials, colchicine may reduce hospital duration (low certainty) and, based on a single small trial, rhG-CSF might reduce mortality and mechanical ventilation in patients with lymphopenia (low certainty).
The authors point out that there is now evidence from several large-scale international trials on remdesivir, hydroxychloroquine, lopinavir/ritonavir, and interferon beta. Unfortunately, however, the trials showed that none of these interventions had a meaningful effect on any patient-important outcomes.
Design
The search strategy and eligibility criteria in this study were comprehensive, without restrictions on language of publication or publication status. In order to make sure that expertise existed in all areas, the research team was composed of clinical and methods experts who have undergone training and calibration exercises for all stages of the review process. Another positive aspect of the study design was that in order to minimise problems with counterintuitive results, the researchers anticipated challenges that would be expected in network meta-analysis when data are sparse. Many of the results for comparison with sparse data were uninformative and sometimes implausible, according to the authors and, therefore, they decided to report evidence on treatments for which at least 100 people were randomised, or where there were at least 20 events. When more data from further treatments become available, the classification of interventions from the most to the least effective will facilitate clear interpretation of results.
Data limitations
The main limitation of the data is that only nine studies were judged to be at low risk of bias. The researchers identify the main setback with the evidence as being the lack of blinding, which might introduce bias through differences in co-interventions between randomisation groups. The researchers chose to consider the treatment arms that did not receive an active experimental drug (ie placebo or standard care) within the same node: it is possible that the unblinded standard care groups received systematically different co-interventions than groups randomised to receive a placebo (Siemieniuk et al, 2020). Direct comparisons in which the evidence is dominated by unblinded studies were rated down, consistent with GRADE, for risk of bias. That is reflected in the rating of the quality of evidence from the network estimate. Much of the gathered data also had reporting concerns. For example, for some outcomes, the method measured and outcomes reported were inconsistent across studies. This led the team to propose a hierarchy for the outcome mechanical ventilation.
The living nature of the study design could potentially (at least temporarily) amplify publication bias, because studies that show promising results are more likely to be published and are published sooner than studies with negative results. Therefore, the inclusion of preprints, many of which have negative results, may reduce this risk. The authors note that industry-sponsored trials such as those for remdesivir and other patented drugs could be particularly at risk of publication bias, and thus positive results for these drugs may need a more cautious interpretation than generic drugs tested in randomised controlled trials independent of industry influence (Siemieniuk et al, 2020
However, the inclusion of preprints in the study by Siemieniuk et al (2020) may also introduce bias from simple errors and the reporting limitations of preprints. They included preprints owing to the urgent requirement for information and because so many of the studies on COVID-19 are published first as preprints. There were no major differences found between preprints and peer-reviewed manuscripts (Siemieniuk et al, 2020).
Another possibility where limitations may have occurred is that the researchers did not detect important subgroup modification. For example, the RECOVERY trial suggested that patients with more severe disease may obtain a greater benefit from dexamethasone than patients with less severe disease. Regardless of this, the proposed subgroup effect has moderate credibility, and the authors state that people reviewing the research should carefully consider the characteristics of the patients included in the trials for each intervention.
Plans for the future
Siemieniuk et al (2020) intend to continue to inform the development of WHO living guidelines and the British Medical Journal Rapid Recommendations. However, they noted an important difference in the methods for assessing the certainty of the evidence. In the living systematic review and network meta-analysis, the authors use a minimally contextualised approach for rating the certainty of the evidence, whereas the guideline panels use a fully contextualised approach, whereby thresholds of importance of magnitudes of effects are dependent on all other outcomes and factors involved in the decision.
The authors are aware of two other similar efforts to their iteration (Boutron et al, 2020; Juul et al, 2020). Siemieniuk et al (2020) made a decision to proceed independently in order to make sure that the results could fully inform clinical decision making for the associated living guidance. They also included a more comprehensive search for the evidence and several differences in analytical methods, which they believe are best suited for this process and acknowledge the importance in evaluating the reproducibility and replicability of results from various scientific approaches.
The researchers plan to periodically update the living systematic review and network meta-analysis, which will aim to contain access to the latest research on the subject and is accompanied by an interactive infographic and regularly updated website (magicapp.org) (Siemieniuk et al, 2020).
Conclusion
The researchers concluded that corticosteroids are likely to reduce mortality, mechanical ventilation, and ventilator-free days in patients with severe COVD-19. Whether or not remdesivir has an impact on any outcome remains uncertain. Hydroxychloroquine, lopinavir/ritonavir, and interferon beta may not reduce mortality or mechanical ventilation, and appear unlikely to have any other benefits. The authors reiterated that the effects of most drug interventions are currently highly uncertain, as no definitive evidence exists that other interventions result in important benefits and harms for any outcomes. It will be interesting to remain abreast of developments as they update their living systematic review and analysis, as a growing plethora of evidence is produced by further trials.