Testing for treatment effect twice using internal and external controls in clinical trials

1.1 Use of external controls in randomized controlled trials (RCTs)

RCTs are the gold standard for generating high-quality causal evidence of new treatments and have long been recognized as the standard method to support key decisions in the drug development process [1,2]. However, despite its clear advantages, the traditional paradigm of conducting RCTs has been increasingly criticized for failing to meet contemporary needs. In certain settings, for example, in HIV prevention [3,4], oncology [5], and neurology [6], randomizing patients to placebo may be difficult for ethical or feasibility reasons. Moreover, adequately powered RCTs are becoming more and more impractical as a growing number of new treatments are targeted toward rare diseases or biomarker-defined subgroups of patients in the era of precision medicine [7].

Meanwhile, a plethora of real-world data (RWD) have been curated for administrative or research purposes and are becoming accessible to researchers in the form of disease registries, administrative claims databases, and electronic health records. These rich data sources can produce valuable insights, i.e., real-world evidence (RWE), into the effect of treatments in routine, daily practice. However, researchers almost ubiquitously caution against possible bias from unmeasured confounding when using RWD.

Being well aware of the limitations of using either RCT or RWD alone, the idea of using RWD to supplement RCT has gained growing interest in recent years. As forcefully argued in the study of Eichler et al. [7], “the future is not about RCTs vs RWE but RCTs and RWE.” There are numerous opportunities in how the integration of RCTs and RWD can achieve fruitful results that using either RCT or RWD alone cannot [8–10]. Among those, an important theme is on augmenting the RCT with RWD to increase efficiency [11–16], and particularly, constructing an externally augmented control arm in the analysis of RCTs [17–20]. Leveraging external controls – relevant individual patient data under control from external trials or RWD – has the potential to reduce the cost of RCTs while increasing the proportion of trial patients given access to novel treatments.

Using external controls is not an entirely new idea. Criteria for evaluating what constitute an acceptable external control arm are proposed in the study of Pocock [21]. It was discussed 20 years ago by the [22, E10 Section 2.5], and also recognized by European Medicines Agency [23], US Food and Drug Administration [24], and National Cancer Institute [25] as one direction to modernize clinical trials. In fact, properly selected external controls (e.g., using propensity score matching) have shown early promise, and several drugs have already been approved based on external control groups [26–28].

Using external controls typically requires the exchangeability condition, i.e., all patient characteristics that affect the potential outcome under control and differ between the trial population and the external control population are measured [29]. While careful adjustment for observed covariates can probably render the exchangeability assumption to hold approximately, the analysis may still be biased due to unmeasured covariates related to “difficulties in reliably selecting a comparable population because of potential changes in medical practice, lack of standardized diagnostic criteria or equivalent outcome measures, and variability in follow-up procedures” [24]. To reduce the potential biases from using external controls, an intuitive frequentist approach is “test-then-pool” that first tests for the comparability of the external controls and internal controls before leveraging external controls [20]. Bayesian methods that rely on power priors have also been popular, which use the likelihood of the external data to a specified power as the prior distribution [30,31]. As such, one can use power priors to adjust the weight allocated to the external information according to the levels of comparability between the external control and the internal data. However, these methods lack formal statistical theory on how the unmeasured biases might affect the validity and efficiency of the proposed procedures.

In this article, we take a different perspective to this problem and propose a sensitivity analysis approach to quantify the magnitude of unmeasured bias that would be needed to alter the study conclusion that presumed no unmeasured biases are introduced by employing external controls [32]. With the unbiased RCT-only test as the benchmark, leveraging external controls increases power or not depends on the interplay between sample sizes and the magnitude of treatment effect and unmeasured biases, which may be difficult to anticipate. This motivates a combined testing procedure that performs both tests, one with and one without external controls, correcting for multiple testing using the joint distribution of the two test statistics. Because the two tests are highly correlated, this correction for multiple testing is small. Interestingly, the proposed combined testing procedure can be viewed as a new method of sensitivity analysis designed for data fusion problems that anchors at the unbiased analysis based on RCT only and “spends” a small proportion of the type I error (i.e., the cost of multiple testing) to also test using the pooled controls. In this way, if leveraging external controls increases power, the power gain compared to the RCT-only test can be substantial; if not, the power loss is small. Before introducing technical details, it is useful to consider a motivating example.

1.2 Example: an RCT in patients with type-2 diabetes

Consider a non-inferiority, phase 3 RCT (referred to as the internal trial, ClinicalTrials.gov number, NCT01894568) comparing a new basal insulin, insulin peglispro, to insulin glargine as the control in Asian insulin-naïve patients with type-2 diabetes using a noninferiority margin of 0.4% [33]. The primary endpoint is the change in hemoglobin A1c (HbA1c) from baseline to 26 weeks of treatment. HbA1c is a continuous-valued measure of average blood glucose in the past 3 months. Before this trial, a phase 3 RCT of similar design (referred to as the external trial, ClinicalTrials.gov number, NCT01435616) has been conducted in the North America and Europe [34], whose control arm will be used as the source of external controls.

We focus on the overweight and obese population, which are, respectively, defined as 23 ≤ body mass index (BMI) < 25 and BMI ≥ 25 for the internal trial according to the Asia-Pacific guidelines, and 25 ≤ BMI < 30 and BMI ≥ 30 for the external trial according to the World Health Organization classifications [35]. There are in total 159 patients under treatment and 150 patients under control in the internal RCT, and 486 patients under control in the external trial. We match 159 similar external controls to the 159 treated patients in the internal RCT using optimal matching based on a robust Mahalanobis distance and a caliper on the propensity score. See Rosenbaum [36, Part II] for discussion of these matching techniques. Table 1 describes covariate balance in the 159 matched pairs. All variables have standardized differences less than 0.13 and are considered sufficiently balanced [37].

Using only the internal RCT, 159 patients under treatment and 150 under control, we conduct a Z -test with the noninferiority margin of 0.4% and obtain a one-sided p -value 7.92 × 1 0 − 7 . In this analysis, the evidence that the new insulin treatment is noninferior to insulin glargine is strong enough when only using the internal controls. On the other hand, under the exchangeability assumption, which implies that the 159 matched external controls are comparable to patients in the internal RCT, we construct an augmented control arm of 309 patients in total and obtain a one-sided p -value 1.88 × 1 0 − 7 . Again, we find strong evidence of noninferiority; however, an investigator may be in doubt about the exchangeability assumption due to the influence of regions on the outcome. Then a natural question is could the one-sided p -value of 1.88 × 1 0 − 7 be due to regions rather than the effect of treatment? If the study conclusion from using external controls can be altered by a plausible effect of regions and because the RCT-only test is already powerful enough, the RCT-only test would be a better choice. However, it would be difficult to know this before examining the data. Motivated by the advice of performing multiple analyses with an appropriate correction for multiple testing given by Rosenbaum [38], we propose a combined testing procedure that performs both analyses, controlling for multiple testing using the joint distribution of the two test statistics. In this article, we will demonstrate that the combined test avoids making an inapt choice about whether to use external controls or not, and only has a small loss of power compared to knowing a priori which is the better choice.

1.3 Outline

Section 2 presents a test that uses only the internal controls and another test that also leverages the external controls, and discusses controlling type I error and comparing power without the exchangeability assumption. Section 3 proposes a combined test that performs both tests and studies in detail its statistical properties. Section 4 presents power calculations. Section 5 returns to the real data applications. Section 6 concludes with a discussion.

2.1 Testing under exchangeability

There is an RCT denoted as D = 1 . Let A be a binary treatment, where A = 1 denotes treatment and A = 0 denotes control, X a vector of observed baseline covariates, Y ( a ) the potential outcome under A = a , for a = 0 , 1 . Throughout the article, we assume consistency and Stable Unit Treatment Value Assumption (SUTVA) so that the observed outcome satisfies Y = A Y ( 1 ) + ( 1 − A ) Y ( 0 ) [39]. Our estimand of interest is the average treatment effect in the RCT population θ ⋆ = E ( Y ( 1 ) ∣ D = 1 ) − E ( Y ( 0 ) ∣ D = 1 ) . In particular, we consider testing a one-sided hypothesis:

The other direction can be considered in the same way. Combining both one-sided tests and applying Bonferroni correction give a two-sided test [40, Section 4.2], and by inversion, a confidence interval.

Write the RCT sample as ( Y i , X i , A i , D i = 1 ) , i = 1 , … , n r , which is assumed to be independent and identically distributed according to the joint law of ( Y ( 1 ) , Y ( 0 ) , X , A ) ∣ D = 1 . Randomization in the RCT guarantees that A ⊥ ( Y ( 1 ) , Y ( 0 ) , X ) ∣ D = 1 and P ( A = a ∣ D = 1 ) = π a > 0 for a = 0 , 1 , with π a known and π 0 + π 1 = 1 . Let Y ¯ a and S a 2 , respectively, be the sample mean and sample variance of the responses Y i ’s from RCT subjects under treatment a , for a = 0 , 1 . Hence, the null hypothesis H 0 can be tested using a simple Z -statistic:

where n 1 and n 0 are, respectively, the number of RCT patients under treatment and control. Based on T 1 , we reject H 0 when T 1 ≥ z 1 − α , where z 1 − α is the ( 1 − α ) th quantile of the standard normal distribution.

To supplement the RCT using external controls, one approach is to first extract external data for patients under control based on the inclusion/exclusion criteria of the RCT and then proceed by matching these external patients to the RCT patients based on their similarity in observed baseline information X [27]. Let D = 0 denote the matched external controls, and thus D = 0 implies A = 0 . Write the matched external controls as ( Y i , X i , A i = 0 , D i = 0 ) , i = 1 , … , n e , which is assumed to be independent and identically distributed according to the joint law of ( Y ( 0 ) , X ) ∣ A = 0 , D = 0 . Suppose that matching has rendered the baseline observed covariates comparable between the RCT and external controls, i.e., D ⊥ X , and that these baseline covariates X explain all differences between the RCT and external controls, i.e., the exchangeability assumption D ⊥ Y ( 0 ) ∣ X holds. This implies D ⊥ ( Y ( 0 ) , X ) and thus E ( Y ( 0 ) ∣ D = 1 ) = E ( Y ( 0 ) ∣ D = 0 ) . Let Y ¯ e be the sample mean of the responses Y i ’s from the external controls, and w Y ¯ 0 + ( 1 − w ) Y ¯ e be a weighted average of mean responses for the two control groups, where w ∈ [ 0 , 1 ] is a pre-specified weight, which could reflect the proportion of the internal control in the two control groups combined. Therefore, the null hypothesis H 0 can also be tested borrowing information from the external controls using

where S e 2 is the sample variance of the responses Y i ’s from external controls. We make two remarks about T 2 ( w ) . First, T 2 ( w ) is constructed assuming independence between the RCT and external controls, which means that T 2 ( w ) may be conservative due to correlation induced by matching [41] but usually to a small extent as the correlation is typically small [42]. Second, T 2 ( w ) , w ∈ [ 0 , 1 ] defines a family of statistics that includes T 2 ( 1 ) = T 1 as a special case. Among those, the exchangeability assumption implies the optimal w that maximizes the efficiency of T 2 ( w ) is proportional to the sample size, i.e., the optimal w equals ( n r π 0 ) ∕ ( n r π 0 + n e ) . One can also choose different values of w to reflect the weights allocated to the two control groups.

2.2 Controlling type I error without exchangeability

The aforementioned approach of leveraging external controls relies on the exchangeability assumption, which may not hold because the RCT patients and external controls may differ with respect to covariates that may not have been measured. Without exchangeability, Y ¯ 1 − { w Y ¯ 0 + ( 1 − w ) Y ¯ e } is not necessarily centered at θ 0 under H 0 and rejecting the null hypothesis when T 2 ( w ) ≥ z 1 − α may inflate type I error.

Define Δ ⋆ = E ( Y ( 0 ) ∣ D = 1 ) − E ( Y ( 0 ) ∣ D = 0 ) , which may be nonzero when exchangeability does not hold. This could occur, for example, if an important prognostic variable is unobserved and left uncontrolled, or if a variable that differs in distribution between D = 0 and D = 1 (such as region) cannot be matched. The correct rejection region for a size- α test based on T 2 ( w ) is

which is infeasible because Δ ⋆ is unknown. To deal with this issue, a tempting choice is to estimate Δ ⋆ and “de-biase” the numerator of T 2 ( w ) to make it mean zero. Nonetheless, estimating Δ ⋆ introduces additional variation, which negatively affects the efficiency of the test. In particular, if one estimates Δ ⋆ by Y ¯ 0 − Y ¯ e , the debiased numerator of T 2 ( w ) becomes Y ¯ 1 − Y ¯ 0 − θ 0 , and the resulting test statistic (after appropriately adjusting for its denominator to reflect the variability of the numerator) becomes equivalent to T 1 , the test statistic without using any external controls.

In order to borrow information from external controls while still controlling type I error, we consider departures from the exchangeability through the lens of a sensitivity analysis [36]. Specifically, we consider a sensitivity parameter Δ 0 such that it bounds the magnitude of bias Δ ⋆ , i.e., Δ 0 ≥ Δ ⋆ . Define

Because Δ ⋆ ≤ Δ 0 , the reject region T 2 , Δ 0 ( w ) ≥ z 1 − α controls type I error at level α . As a special case when Δ ⋆ ≤ Δ 0 holds with Δ 0 = 0 (e.g., under exchangeability), T 2 , Δ 0 ( w ) ≥ z 1 − α becomes T 2 ( w ) ≥ z 1 − α , the reject region under exchangeability. As Δ 0 increases, there is greater uncertainty about how the exchangeability might be violated, leading to more stringent rejection criterion to control type I error. The reject region T 2 , Δ 0 ( w ) ≥ z 1 − α is sharp under Δ ⋆ ≤ Δ 0 in the sense that they are of size- α when Δ ⋆ = Δ 0 , so it cannot be improved unless further information is provided.

2.3 Power comparison without exchangeability

Write σ a 2 = Var ( Y ( a ) ∣ D = 1 ) , for a = 0 , 1 , and σ e 2 = Var ( Y ( 0 ) ∣ D = 0 ) . Under the alternative hypothesis H A : θ ⋆ > θ 0 , the power of T 1 is the probability of event T 1 ≥ z 1 − α , which is asymptotically equal to

where Φ ( ⋅ ) is the standard normal cumulative distribution. In parallel, the power of T 2 , Δ 0 ( w ) is the probability of event T 2 , Δ 0 ( w ) ≥ z 1 − α , which is asymptotically equal to

Several remarks are in order based on the above power formulas. First, the power of T 2 , Δ 0 ( w ) is larger than that of T 1 if and only if

For instance, when Δ 0 = Δ ⋆ , i.e., the specified upper bound for Δ ⋆ is tight, and σ 0 2 = σ e 2 , i.e., the variance of Y for the two control groups are equal, simple algebra reveals that the power of T 2 , Δ 0 ( w ) is always larger than that of T 1 for any w satisfying max ( 0 , ( n r π 0 − n e ) ∕ ( n r π 0 + n e ) ) ≤ w < 1 .

Second, we can derive the oracle w that maximizes the power of T 2 , Δ 0 ( w ) . Let κ = ( π 0 − 1 σ 0 2 ) ∕ ( π 1 − 1 σ 1 2 + π 0 − 1 σ 0 2 ) , the optimal w takes the following form:

where the first case is when Δ 0 is specified too large, the power of T 2 , Δ 0 ( w ) is maximized at w = 1 , which means that using the external controls does not lead to efficiency gain. As an illustration, under the special case that π 1 − 1 σ 1 2 = π 0 − 1 σ 0 2 = n r n e − 1 σ e 2 , when Δ 0 − Δ ⋆ > ( θ ⋆ − θ 0 ) ∕ 2 , the optimal w is 1, whereas when ( θ ⋆ − θ 0 ) ∕ 2 > Δ 0 − Δ ⋆ > 0 , the optimal w is 1 − { ( θ 0 − θ ⋆ ) + 2 ( Δ 0 − Δ ⋆ ) } ∕ { 2 ( θ 0 − θ ⋆ ) + ( Δ 0 − Δ ⋆ ) } . Under another special case when Δ ⋆ = Δ 0 and σ 1 = σ 0 = σ e , w opt becomes ( n r π 0 ) ∕ ( n r π 0 + n e ) , which agrees with the optimal w under exchangeability discussed in Section 2.1. The proof of (3) is given in the supplementary material.

Finally, we compare the two tests T 1 and T 2 , Δ 0 ( w ) in terms of their limiting power as the sample sizes grow to infinity. When θ ⋆ > θ 0 and lim n r → + ∞ n r ( θ ⋆ − θ 0 ) = + ∞ (e.g., when θ ⋆ , θ 0 are two constants), then the power of T 1 goes to 1 as n r → ∞ . In contrast, the limiting power of T 2 , Δ 0 ( w ) depends on specifications of w and Δ 0 . Specifically, as can be seen from the power formula in (2), when min ( n r , n e ) → + ∞ , the power of T 2 , Δ 0 ( w ) tends to 1 if θ 0 − θ ⋆ + ( 1 − w ) ( Δ 0 − Δ ⋆ ) < 0 , i.e., when Δ 0 < ( θ ⋆ − θ 0 ) ∕ ( 1 − w ) + Δ ⋆ , and to 0 if θ 0 − θ ⋆ + ( 1 − w ) ( Δ 0 − Δ ⋆ ) > 0 , i.e., when Δ 0 > ( θ ⋆ − θ 0 ) ∕ ( 1 − w ) + Δ ⋆ . In other words, there exists a w -dependent number Δ ˜ ( w ) = ( θ ⋆ − θ 0 ) ∕ ( 1 − w ) + Δ ⋆ that characterizes the limiting behavior of T 2 , Δ 0 ( w ) under the alternative: the power of T 2 , Δ 0 ( w ) tends to 1 if Δ 0 < Δ ˜ ( w ) and to 0 if Δ 0 > Δ ˜ ( w ) as min ( n r , n e ) → + ∞ . In plain language, since the maximum bias Δ 0 counteracts with θ ⋆ − θ 0 , the difference we would like to detect, when Δ 0 exceeds a certain threshold Δ ˜ ( w ) , the maximum bias starts to dominate the true difference θ ⋆ − θ 0 , resulting in no power to detect the difference. This number Δ ˜ ( w ) is analogous to the design sensitivity in the literature of sensitivity analysis [36,43].