Outcome Reporting Bias in COVID-19 mRNA Vaccine Clinical Trials

Outcome Reporting Bias in COVID-19 mRNA Vaccine Clinical Trials

Relative risk reduction and absolute risk reduction measures in the evaluation of clinical trial data are poorly understood by health professionals and the public. The absence of reported absolute risk reduction in COVID-19 vaccine clinical trials can lead to outcome reporting bias that affects the interpretation of vaccine efficacy. The present article uses clinical epidemiologic tools to critically appraise reports of efficacy in Pfizer/BioNTech and Moderna COVID-19 mRNA vaccine clinical trials. Based on data reported by the manufacturer for Pfizer/BioNTech vaccine BNT162b2, this critical appraisal shows: relative risk reduction, 95.1%; 95% CI, 90.0% to 97.6%; p = 0.016; absolute risk reduction, 0.7%; 95% CI, 0.59% to 0.83%; p < 0.000. For the Moderna vaccine mRNA-1273, the appraisal shows: relative risk reduction, 94.1%; 95% CI, 89.1% to 96.8%; p = 0.004; absolute risk reduction, 1.1%; 95% CI, 0.97% to 1.32%; p < 0.000. Unreported absolute risk reduction measures of 0.7% and 1.1% for the Pfzier/BioNTech and Moderna vaccines, respectively, are very much lower than the reported relative risk reduction measures. Reporting absolute risk reduction measures is essential to prevent outcome reporting bias in evaluation of COVID-19 vaccine efficacy. See the original article from Medicina here.

Introduction

Using messenger RNA (mRNA) in vaccines to produce proteins that trigger an immune response against infectious diseases has held promise for decades, but until recently, no clinically tested mRNA vaccine has managed to advance beyond small, early-phase trials [1]. Normally, genetic code in mRNA is transcribed from DNA in the cell nucleus, and the coded message is delivered by mRNA to cell ribosomes for translation during protein biosynthesis [2]. COVID-19 mRNA vaccines directly inject cells with a synthetic genetic code to replicate the spike S protein found on the surface of the coronavirus, SARS-CoV-2 [3]. Once replicated, the spike protein is proposed to trigger an immune response that creates antibodies against the virus [4].

However, several biological obstacles continue to challenge the development of mRNA vaccines, including “mRNA’s extremely large size, charge, intrinsic instability, and high susceptibility to enzymatic degradation” [5]. To mitigate enzymatic degradation, mRNA in the vaccines is encapsulated in lipid nanoparticles [6], but it is unclear how this encapsulation affects genetic code translation in the cell ribosomes. Nevertheless, clinical results of phase III trials reported for COVID-19 vaccines manufactured by Pfizer/BioNTech (New York City, NY, USA/Mainz, Germany) [7] and Moderna (Cambridge, MA, USA) [8] have far surpassed predicted performance, with vaccine efficacy rates of approximately 95%. Curiously, “why these vaccines seem so effective while previous attempts against other pathogens haven’t appeared as promising remains an open question” [1].

As noted in BMJ Opinion, 26 November 2020 [9],“There may be much more complexity to the ‘95% effective’ announcement than meets the eye—or perhaps not. Only full transparency and rigorous scrutiny of the data will allow for informed decision making. The data must be made public.”

As was also noted in the BMJ Opinion, Pfizer/BioNTech and Moderna reported the relative risk reduction of their vaccines, but the manufacturers did not report a corresponding absolute risk reduction, which “appears to be less than 1%” [9]. Absolute risk reduction (ARR) and relative risk reduction (RRR) are measures of treatment efficacy reported in randomized clinical trials. Because the ARR and RRR can be dramatically different in the same trial, it is necessary to include both measures when reporting efficacy outcomes to avoid outcome reporting bias. In the present article, a critical appraisal of publicly available clinical trial data verifies that absolute risk reduction percentages for Pfizer/BioNTech vaccine BNT162b2 [7] and Moderna vaccine mRNA-1273 [8] are, respectively, 0.7%; 95% CI, 0.59% to 0.83%; p = 0.000, and 1.1%; 95% CI, 0.97% to 1.32%; p = 0.000. The same publicly available data, without absolute risk reduction measures, were reviewed and approved by the roster of members serving on the U.S. Food and Drug Administration’s (FDA’s) Vaccines and Related Biological Products Advisory Committee (VRBPAC) for emergency use authorization (EUA) of the mRNA vaccines [10]. Ironically, the omission of absolute risk reduction measures in data reviewed by the VRBPAC overlooks FDA guidelines for communicating evidence-based risks and benefits to the public [11]. The FDA’s advice for information providers includes:

“Provide absolute risks, not just relative risks. Patients are unduly influenced when risk information is presented using a relative risk approach; this can result in suboptimal decisions. Thus, an absolute risk format should be used.”

The New England Journal of Medicine also published clinical trial data on safety and efficacy for the BNT162b2 vaccine [12] and the mRNA-1273 vaccine [13], but with no mention of absolute risk reduction measures.

The present article uses epidemiologic tools to critically appraise absolute and relative risk reduction measures for vaccine efficacy in phase III clinical trials of the COVID-19 mRNA vaccines. Microsoft Excel was used to analyze data and chart risk reduction outcomes. The article further clarifies how selective reporting of vaccine efficacy measures can cause a type of outcome reporting bias that misrepresents health information disseminated to the public.

Critical Appraisal of Vaccine Efficacy

The application of epidemiologic and biometric methods to clinical diagnosis and treatment is known as clinical epidemiology [14]. Clinical epidemiologic tools can be applied in evidence-based medicine (EBM) to critically appraise research evidence for validity, size of effect, and usefulness in clinical practice [15]. Clinical treatment effects in groups of participants are measured by comparing probabilities of an event, known as event rates [16].

Figure 1 shows an example of a vaccine clinical trial for an infectious disease. The vaccine and placebo groups in Figure 1 each have 100 randomly assigned individuals with no history of infection, and an event is defined as the incidence of infection among all individuals during the course of the trial. The percentage of events in the vaccine group is the experimental event rate (EER) or the risk of infection in the vaccine group (1/100 = 1%), and the percentage of events in the placebo group is the control event rate (CER) or the risk of infection in the placebo group (2/100 = 2%). Absolute risk reduction (ARR) is the disease risk difference between the placebo and vaccine groups, i.e., the CER minus the EER (2% − 1% = 1%). The ARR is also known as the vaccine disease preventable incidence (VDPI) [17]. Relative risk reduction (RRR) or vaccine efficacy (VE) is the reduced risk from vaccination, the ARR or VDPI, relative to or divided by the risk in unvaccinated individuals, the CER (1%/2% = 50%) [18].

Medicina 57 00199 g001 550

Discussion

Medical and public health experts continue to stress the need to include measurements of absolute risk reduction and number needed to treat when reporting results of clinical interventions [28]. Currently, differences between relative effect measures and absolute effect measures in studies are “poorly understood by health professionals, and even more poorly understood by patients.” [29]

In addition,“…critical appraisal knowledge and skills are limited among physicians,” and “use of relative effect measures was associated with greater perceptions of medication effectiveness and intent to prescribe, compared with the use of absolute effect measures.”[29]

Reporting relative measures may be sufficient to summarize evidence of a study for comparisons with other studies, but absolute measures are also necessary for applying study findings to specific clinical or public health circumstances [22]. Omitting absolute risk reduction findings in public health and clinical reports of vaccine efficacy is an example of outcome reporting bias, which ignores unfavorable outcomes and misleads the public’s impression and scientific understanding of a treatment’s efficacy and benefits [30]. Furthermore, the ethical and legal obligation of informed consent requires that patients are educated about the risks and benefits of a healthcare procedure or intervention [31].

Similar to the critical appraisal in the present article, critical appraisals of reported vaccine efficacy in other studies reveals clinically significant insights. For example, a 2018 review of 52 randomized trials for influenza vaccines that studied over 80,000 healthy adults reported an overall influenza vaccine EER of 0.9% and a 2.3% CER, which calculates to a RRR of 60.8% [32]. This vaccine efficacy is consistent with a 40% to 60% reduction in influenza reported by the Centers for Disease Control and Prevention (CDC) [33]. However, critically appraising data from the 2018 review shows an overall ARR of only 1.4%, which reveals vital clinical information that is missing in the CDC report. A 1.4% ARR works out to a NNV of approximately 72 people, meaning that 72 individuals need to be vaccinated to reduce one case of influenza. By comparison, Figure 2 of the present article shows that the NNV for the Pfizer-BioNTech and Moderna vaccines are 142 (95% CI 122 to 170) and 88 (95% CI 76 to 104), respectively.

The mRNA vaccine manufacturers reported that infections in most subgroups in phase III clinical trials were similar for both vaccines after two doses. Vaccine clinical trial case definitions for SARS-CoV-2 infection included COVID-19 clinical symptoms; thus the trials were not designed to provide evidence of vaccine efficacy for protection against asymptomatic infections. In addition to outcome reporting bias, information bias may have also affected COVID-19 vaccine trial outcomes due to misclassification of SARS-CoV-2 infections as mild adverse effects of the vaccines. For example, several COVID-19 clinical symptoms are similar to the vaccines’ adverse effects such as fever, pain, and fatigue, which could potentially lead to missed diagnoses of viral infections.

A limitation of this article is that it only critically appraised mRNA vaccine efficacy in healthy individuals who were randomized to two groups under strictly controlled conditions. The critical appraisal did not include vaccine safety and effectiveness outcomes within a general population that includes unhealthy people and that lacks control over confounding factors. For example, healthy vaccinee bias occurs when people who are in better health are more likely to follow vaccination recommendations in order to protect their health [34].

Conclusions

A critical appraisal of phase III clinical trial data for the Pfizer/BioNTech vaccine BNT162b2 and Moderna vaccine mRNA-1273 shows that absolute risk reduction measures are very much lower than the reported relative risk reduction measures. Yet, the manufacturers failed to report absolute risk reduction measures in publicly released documents. As well, the U.S FDA Advisory Committee (VRBPAC) did not follow FDA published guidelines for communicating risks and benefits to the public, and the committee failed to report absolute risk reduction measures in authorizing the BNT162b2 and mRNA-1273 vaccines for emergency use. Such examples of outcome reporting bias mislead and distort the public’s interpretation of COVID-19 mRNA vaccine efficacy and violate the ethical and legal obligations of informed consent.

Responses

This site uses Akismet to reduce spam. Learn how your comment data is processed.

    1. Reminds me a little of auditors doing risk assessments, and reporting a risk as the "absence of a control". It’s only a risk if it’s a control that addresses an event that is likely to occur.
      IMHO, NIH statement/quotation is being taken out of context. As you did, I would ask the author to describe how he would gather absolute risk data for COVID.

      1. By measuring IFR (Infection Fatality Rate) through antibody testing in the given population. It’s been done at many places for COVID and IFR is between around 0.3% on average, greatly varying by factors like place, age, or comorbidities.

  1. First of all, calculating absolute risk and relative risk are straightforward providing you have the data. e.g. If you want to understand absolute risk in a population as a whole, you would need a pretty good cross section but does it have any meaning in a populace that wears face masks and stays home v. front line workers?
    Secondly, the studies clearly state relative risk. Not reporting absolute risk data is an omission that can be justified but it is not bias.

  2. I’m disappointed in TSN on this one. The absolute risk is a function of how prevalent the disease was in the population at the time. Using relative risk, in this case, is really the only meaningful thing that can be taken away from such a trial. It’s almost like TSN is playing the same idiotic game of headline grabbing as "the rest of them", which makes me rather sad.