Estimation and interpretation of vaccine efficacy in COVID-19 randomized clinical trials

Abstract Objectives An exceptional effort by the scientific community has led to the development of multiple vaccines against COVID-19. Efficacy estimates for these vaccines have been widely communicated to the general public, but are nonetheless challenging to compare because they are based on phase 3 trials that differ in study design, definition of vaccine efficacy and the handling of cases arising shortly after vaccination. We investigate the impact of these choices on vaccine efficacy estimates, both theoretically and by re-analyzing the Janssen and Pfizer COVID-19 trial data under a uniform protocol. We moreover study the causal interpretation that can be assigned to per-protocol analyses typically performed in vaccine trials. Finally, we propose alternative estimands to measure the intrinsic vaccine efficacy in settings with delayed immune response. Methods The data of the Janssen COVID-19 trials were recreated, based on the published Kaplan-Meier curves. An estimator for the alternative causal estimand was developed using a Structural Distribution Model. Results In the data analyses, we observed rather large differences between intention-to-treat and per-protocol effect estimates. In contrast, the causal estimand and the different estimators used for per-protocol effects lead approximately to the same estimates. Conclusions In these COVID-10 vaccine trials, per-protocol effects can be interpreted as the number of cases that can be avoided by vaccination, if the vaccine would immediately induce an immune response. However, it is unclear whether this interpretation also holds in other settings.