Estimation and interpretation of vaccine efficacy in COVID-19 randomized clinical trials

Per-protocol effects in COVID-19 trials

The aim of per-protocol analyses is to obtain insight into the intrinsic efficacy of the vaccine, i.e. the relative reduction in infections due to vaccination in fully compliant subjects, after completion of the prescribed vaccination regimen and achieving adequate immune response (Horne, Lachenbruch, and Goldenthal 2000; Hudgens, Gilbert, and Self 2004). However, this approach can negatively impact power relative to including all cases observed since baseline, especially in settings with low disease incidence (Dean, Halloran, and Longini 2018). It is moreover vulnerable to selection bias as per-protocol analyses entail comparison of subgroups selected post randomization (Hudgens et al., 2004).

In the Janssen trial, the cumulative incidence in the full analysis set was similar in both the vaccine and placebo arm until around day 14, after which the curves diverged, with more cases accumulating in the placebo group than the vaccine group (Figure 1). Therefore, we expect the possible selection bias in the per-protocol analysis to be more severe when cases observed before day 28 are removed compared to when cases before day 14 are removed. Selection bias may occur if cases develop during the ramp-up period and vaccinated and placebo groups are no longer at comparable risk of infection after this period (Horne et al. 2000). In the Janssen trial, approximately 77 (18%) of the placebo cases and 75 (39%) of the vaccine cases were observed during the first 14 days, even though it is expected that only a small proportion of the total cases occurs shortly after randomization (Horne et al. 2000). If the vaccine has no effect on infections and there are no side effects during the ramp-up period, it is expected that removing cases observed during this period does not introduce selection bias, as these cases are then comparable across arms. One then obtains the vaccine effect for a subgroup of the study population so that the per-protocol effect may nevertheless differ from the intention-to-treat (ITT) effect that takes into account all cases after randomization (Appendix C.1).

Figure 1:

Recreated data similar to the COVID-19 vaccine trials conducted by Pfizer (Food and Drug Administration 2020c) and Janssen (Food and Drug Administration 2021). The grey vertical lines indicate the visits at which a dose of the vaccine is given and the pink line indicates the end of the ramp-up period, as specified in the study protocol.

Other pharmaceutical companies also included some time lag beyond completion of the last vaccination dose to allow for optimal immunity (Table 1). In the Pfizer trial, the cumulative incidence in the full analysis set was again similar in both arms until approximately 14 days after randomization, at which time point the survival curves diverged (Figure 1). However, a ramp-up period of 28 days (after randomization) was specified, which resulted in the removal of approximately 94 (34%) of the placebo and 41 (82%) of the vaccine cases. For the AstraZeneca and Moderna trials, it also appears that the vaccine already had a clear effect on infection before the end of the specified ramp-up period (Figure 7 in European Medicines Agency (2021) and Figure 2 in Food and Drug Administration (2020b)).

In the Moderna trial, vaccine efficacy was estimated using a hazard ratio that was obtained by fitting a stratified Cox proportional hazards model (Cox 1972), where patients who got infected before day 14 after the second dose were censored (Food and Drug Administration 2020b). This approach is also subject to a possible selection bias, since the implicit assumption of non-informative censoring is violated as early cases are censored based on their infection time. None of the aforementioned pharmaceutical companies reported a rationale for the choice of the length of the ramp-up period in the protocol (European Medicines Agency 2021; Food and Drug Administration 2020b, 2020c, 2021), even though it is recommended by the WHO (World Health Organization 2020a).

The WHO (World Health Organization 2020a) argued that, in general, the ITT estimate will tend to be diluted compared to the PP vaccine efficacy estimates, since individuals typically fail to comply with the protocol for reasons related to the vaccine itself (Dean et al. 2019). In addition, including cases that arise during the ramp-up time will typically lead to smaller vaccine efficacy estimates since the vaccine is not yet fully effective during this period. Since ITT vaccine efficacy estimates provide information about the effectiveness of a public health strategy using the vaccine, and because they reflect the speed at which the vaccine becomes protective, they may be more meaningful than per-protocol effects to compare vaccines with different dosing regimens or ramp-up periods (World Health Organization 2020a). Therefore, it has been recommended to report both vaccine efficacy estimates (Horne et al. 2000).

Hypothetical vaccine efficacy estimands

In this section, we propose two new estimands that can be used to measure vaccine efficacy in settings with delayed immune response and give insight into the intrinsic effect of the vaccine after achieving adequate immune response. In addition, we discuss new estimators for these estimands and clarify under what assumptions they can be approximated by standard per-protocol estimators.

Vaccine efficacy if infections during ramp-up can be prevented

First, we consider the vaccine efficacy that would have been observed if cases during the ramp-up period could have been avoided. This is an example of a hypothetical estimand (International Council for Harmonisation 2019) and might be of particular interest since it is an effect that can be realized in practice for several viruses. For example, influenza vaccines cause antibodies to develop in the body about two weeks after vaccination (Centers for Disease Control and Prevention 2021); influenza infections during this period can be avoided by vaccinating people early enough, i.e. at least two weeks before flu season begins. Further, vaccines against diseases endemic to certain countries, e.g. Malaria in Africa (Centers for Disease Control and Prevention 2020), are given early enough before traveling, allowing the immune response to be developed before arriving in these countries. In the COVID-19 setting, cases occurring shortly after vaccination can be avoided by quarantine. In general, we consider a setting where cases during the ramp-up period can be avoided.

It is not immediately clear how this effect can be identified without relying on strong assumptions. In particular, the observed infection hazard ratios after the ramp-up period cannot simply be used as substitution for the infection hazard ratio in the hypothetical setting because the populations not at risk for infections at a given time are likely not exchangeable between the observed and hypothetical setting. In Appendix C.2, we show that the per-protocol estimator, which removes cases observed during the ramp-up time, is unbiased for this effect only under strong assumptions. In particular, one must assume that patients who are infected during the ramp-up time would have comparable infection times as patients who were not infected during this period, if infections during this period could have been avoided. This is likely implausible as early cases may well be selective.

Vaccine efficacy if ramp-up period can be eliminated

Next, we consider the vaccine efficacy, if the vaccine would immediately induce an immune response; i.e. if there was no ramp-up period. This estimand provides insight into the intrinsic effect of the vaccine because it represents the effect once subjects are fully immunized.

In Appendix C.3, we propose an estimator for this hypothetical VE which relies on a Structural Distribution Model (SDM) for identification (Robins 1994; Vansteelandt and Joffe 2014). This model maps percentiles of the distribution of infection times under placebo into percentiles of the distribution of infection times under vaccine. It makes assumptions about the effect of the vaccine on the infection times on population level, but we refrain from making assumptions on individual infection times. In particular, if the vaccine would work immediately, we would impose that vaccination multiplies the quantiles of the infection distribution by a factor exp(ψ), for a scalar ψ. However, we assume that the vaccine effect is limited during the ramp-up time, and therefore, the quantiles are multiplied by a (possibly) smaller factor exp(ρψ) with ρ ∈ [0, 1]. Formally, let α denote the length of the ramp-up time, S(t|X=0) the survival function at time t in the placebo arm and S _SDM(t|X=1; α, ρ, ψ) the modeled survival function at time t in the vaccine arm. If we assume that the vaccine is fully effective after the ramp-up period, this leads to model

In this model, ψ represents the vaccine effect, with higher values indicating higher efficacy. Parameter ρ indicates how much weaker the vaccine effect is during the ramp-up time than after. The choice ρ=0 expresses no vaccine effect during the ramp-up period, while ρ=1 indicates full vaccine effect from baseline (Figure 2). Model (1) allows to impose that the vaccine has a different effect during and after the ramp-up time. The parameters can be estimated by comparing the mapped survival function (1) to the observed survival function in the vaccine arm. The hypothetical vaccine efficacy can then be estimated by setting the ramp-up period to 0 days in model (1) (or ρ to 1). Details about this model and estimation can be found in Appendix C.3. Other models than model (1) could be used, e.g. one could impose that the effect of the vaccine increases during the ramp-up time by assuming that the quantiles of infection during the ramp-up time are multiplied by exp (ψ(1 + (t − α)/α)). However, in the remainder of the paper we will focus on model (1).

Figure 2:

Illustration of the impact of parameter ρ on the cumulative incidence in model (1) for fixed ψ=0.8 and α=20. Plots are based on simulated data.

Data

The data of the Janssen (ClinicalTrials.gov NCT0450572) and Pfizer (ClinicalTrials.gov NCT04368728) COVID-19 trials were recreated, based on the published Kaplan-Meier curves (Figure 1 in Food and Drug Administration (2021) and Figure 2 in Food and Drug Administration (2020c)), as described in Appendix C.4.1. Figure 1 visualizes the obtained Kaplan–Meier curves, which agree very well with the published curves. The R-code used to create these datasets is provided in Appendix D.1.

Methods

For both trials, the ITT and PP vaccine efficacy effects were estimated every week for the entire study duration, using the cumulative incidence, the hazard and incidence rate as risk measures. ITT effects included all cases observed since randomization, while for the PP effects cases observed during the ramp-up time were removed or censored. Different lengths of ramp-up times (α=7, 14, 28 and 35 days) were investigated for both trials. Hazard ratios were estimated by fitting a Cox proportional hazards model (Cox 1972) and cumulative incidences were obtained by estimating Kaplan–Meier curves (Kaplan and Meier 1958). Incidence rates were acquired by fitting a Poisson model (Nauta 2010) with the logarithm of the observation time as offset to account for follow-up time. All estimators assume censoring to be non-informative within each treatment arm. The hypothetical estimand “if the ramp-up period could be eliminated” was also estimated every week, using the cumulative incidence as risk measure, as described in Appendix C.4.3. R-code for these estimators is provided in Appendix D.2. Standard errors (SE) were obtained using 1,000 non-parametric bootstrap replications (Efron 1979).

Results

Table 2 shows the obtained vaccine efficacy estimates and SEs for the Pfizer and Janssen COVID-19 vaccine trials, using the ramp-up period as specified in the study protocols. For both trials, the ITT effect estimates are approximately 10% smaller than the PP effects. This is in contrast with the results of Horne et al. (Horne et al. 2000) who generally observed little difference between ITT and PP effects. However, these authors also found a few trials reported in the last 20 years where efficacy estimates under the two approaches gave discordant results.

In addition, PP effects where cases observed during the ramp-up period are removed versus censored coincide in our results. However, the vaccine efficacy estimates differ by the risk measure used, even though that was not expected because of the small incidence rates (Hudgens et al. 2004). Moreover, for most effects, the obtained standard errors were two to three times larger when using the cumulative incidence compared to the hazard or incidence rate. This can be attributed to the (semi-)parametric nature of the Cox and Poisson model but comes at the risk of bias when this model is not correct. Figure 3 shows that the effect estimates converge over time. In the Janssen trial, an increase in VE was noticed 84-91 days after vaccination when using the cumulative incidence as risk measure. This is probably because many patients are censored during that period (Figure 1) and is not observed when using the other risk measures.

Figure 3:

Vaccine efficacy estimates ±2SE for the Pfizer and Janssen dataset are shown over time, both for the cumulative incidence, the hazard and incidence rate as risk measure. Every point represents the VE estimate that would be obtained if the trial were stopped at the corresponding visit and all information up till that visit was used.

Tables 1 and 2 and Figures 10 and 11 in Appendix C.4.2 show the results when using other lengths of the ramp-up period. From these results, it turns out that choosing too short a period is more problematic than too long a period. In particular, when α is set to 7 days in the Janssen trial and the Pfizer trial, diluted vaccine efficacy estimates are obtained compared to the original PP effects. In both trials, the PP effects converge to approximately the same limit when specifying a ramp-up period of 14, 28 or 35 days. However, in the Janssen trial, the obtained SEs are somewhat larger when more cases are removed/censored.

Table 2 also shows estimates for the hypothetical estimand “if there was no ramp-up time”. Although different estimators were used (Appendix C.4.3), we only show results for the one-parameter model method with the ramp-up period specified as in the study protocol. The obtained effect estimates are very close to the PP estimates with only a slightly larger SE (Table 2). For the Pfizer trial, the Structural Distribution Model fitted the observed data very well (Figures 13–15 in Appendix C.4.3), but less so for the Janssen study (Figures 17 and 18 in Appendix C.4.3).