On Stratified Adjusted Tests by Binomial Trials

1 Introduction

In several medical studies conducted to test a treatment effect, such as a new drug or operative method, the response is frequently given as a binary endpoint such as “success” vs. “failure”. The two groups compared are frequently referred to as the “treatment” and “control” groups. Then, the results of the study are summarized in a 2×2 contingency table that is dichotomous by treatment and result.

In practice, there is also a covariate that will likely affect the association between the treatment or control group and the response. When there is an imbalance of the covariate between the two groups, evaluating the overall treatment effect while ignoring this imbalance would lead to a biased result. This issue is equally relevant when considering either a multicenter clinical trial or a meta-analysis. That is, there is a possibility that the potential background factors of patient, treatment environment, or medical skills differ between each center or trial. To adjust for such imbalances, stratified analyses or analyses based on an appropriate regression model such as logistic regression are widely used [1, 2]. In this study, we focus on the use of a stratified analysis.

The most commonly used measures of a treatment effect are relative risk, odds ratio, and risk difference. The advantages and disadvantages of these measures are described in Lachin [1] and Fleiss et al. [3]. Although the following discussion may also be relevant when considering any criteria, we specifically consider the case based on risk difference for the sake of simplicity and to allow for natural interpretation. If the true success probabilities of the treatment and control groups in stratum i are represented as pTi and pCi, respectively, then the risk difference in the stratum is defined as δi=pTi−pCi. Thus, the true overall treatment effect is expressed as δ=∑iπiδi, where πi is the true ratio of patients that would have entered stratum i in the population [4].

To examine the statistical significance of the true overall treatment effect under the case in which the strata are imbalanced, several approximation test statistics of the hypothesis H0:δ=0 are proposed. As a simple approach based on the weight of the reciprocal of the squared standard error of δi, the weighted mean divided by its standard error can be used. This statistic approximately follows the chi-square distribution with 1 degree of freedom under H0 [3]. However, it is also known that the performance of this approach is not good in many situations. The most commonly used approaches are the Cochran test [5] and Mantel-Haenszel test [6]. There are several ways to derive these test statistics; for example, these can be viewed as summarized statistics based on risk differences weighted by the harmonic mean of the number of patients in each stratum [7]. From another perspective, the Cochran test can be viewed as a test based on two independent groups according to the binomial distribution in each stratum [1]. The Mantel-Haenszel test can be considered to be based on the hypergeometric distribution [1]. The results of these tests are often very similar, and quickly converge to the same value. Yusuf et al. [8] proposed a Mantel-Haenszel test with the continuity correction term removed. A comparison of these methods based on simulation studies is described in Sánchez-Meca and Mar\’in-Martńez [7]. Mehrotra and Railkar [4] proposed another test method that could minimize the squared error between the true overall treatment effect and the estimated treatment effect. A detailed discussion of this approach to a treatment by stratum interaction is provided by Mehrotra [9].

As a more general framework, many estimators that adjust for covariates in randomized trials (to estimate the overall treatment effect) are proposed. The comparative research on the estimators that are focused on the binary outcome cases was published by Colantuoni and Rosenblum [10]. In their research, the seven estimators are studied in terms of the properties and performance based on simulations. Although these estimators can be applied to the case of the randomized stratification, as discussed in their paper, we consider simpler cases in this study. That is, we consider the case where the independence of the outcome and covariates (that are not used in the stratification) is satisfied.

There are two types of sampling structures for a stratified analysis in a randomized clinical trial: random assignment of a treatment within each stratum, and completely randomized assignment of the treatment. If the covariates that are likely to affect the association between the treatment or control group and the response are known at the start of the trial, random assignment of the treatment within each stratum would be considered. On the other hand, if these covariates are not clear at the start of the trial, or if it is difficult to allocate the treatment within each stratum, completely randomized assignment of the treatment would be performed. Although the debate as to which structure is preferable continues, in both sampling structures, the stratified adjusted tests described above are generally used to assess the significance of the overall treatment effect. Following the terminology employed by Ganju and Zhou [11], we refer to the random assignment of a treatment within each stratum as pre-stratification, and the completely randomized assignment is referred to as post-stratification.

Although the stratified adjusted tests such as the Cochran test show good performance for assessing the significance of an overall treatment effect in a stratified analysis, these tests are constructed based on the assumption that the number of patients within a stratum is fixed. However, in practice, the stratum sizes are not fixed at the start of the trial in many situations, and are instead allowed to vary. For example, in a multicenter trial using a pre-stratification design, the total sample size might be fixed but it is nevertheless difficult to control the sample size in each stratum in many cases [12]. In a randomized clinical trial using the post-stratification design, it would be impossible to fix the number of patients in each stratum (such as sex and age) before the start of the clinical trial. Therefore, there is a risk that using such tests under these situations will lead to potentially ignoring the variance in the number of patients in each stratum. Consequently, the variation in the test statistics is considered underestimated in many situations.

To handle this problem, we propose new test statistics applicable for the pre-stratification and post-stratification designs. In our approach, the number of patients in the strata are assumed to follow a multinomial distribution. This assumption is reasonable for many randomized clinical trials. In fact, [12] proposed a test statistic under this same assumption for evaluating the significance of the overall treatment effect using continuous data in multicenter trials with post-stratification. With this assumption, the number of success patients in each stratum is viewed as a result of a two-step process. That is, the number of patients in a stratum is given by following the multinomial distribution in the first stage, and then the number of success patients for the treatment in each stratum is given by following the binomial distribution. From another perspective, the number of success patients in each treatment can also be considered to be given by a finite mixture model.

Since we assume that the two independent groups have the binomial distributions in each stratum, our proposed approach is based on the Cochran test. The difference between the proposed test and ordinary Cochran test is given by the difference in the estimator of the variance of the overall treatment effect. When the total number of patients is small, this variance estimator of our proposed approach tends to become larger than that of the Cochran test because the variability of the estimator of the treatment effect in each stratum becomes large or the number of patients included in the two groups tends to become imbalanced in each stratum. This variability or imbalance is reduced by increasing the total number of patients, and as a result, our proposed approach and the Cochran test yield similar values.

In our proposed approach, the variation of the estimated treatment effect is expected to be higher than that obtained with the traditional approach in many situations. This fact seems to be similar to the approach considering a random-effects model for combining the evidence of several studies to compare two treatments in a meta-analysis [13, 14]. Although the results of these two approaches show trends in the same direction, their basic underlying ideas and assumptions are distinct. In the random-effects approach, the model assumes random variation among the strata with respect to the expected treatment effect, and thus the estimated variation of the statistics is higher than that obtained with a fixed-effects approach. In our approach, on the other hand, the expected treatment effect is fixed in each stratum. The change in the variation of the estimated treatment effect is instead due to the variation in the number of patients in each stratum.

The remainder of this paper is organized as follows. In Section 2, we introduce the notation and traditional stratified adjusted test used to assess the significance of an overall treatment effect. In Section 3 and Section 4, the new approaches considering the variation in the number of patients in the strata are described for the pre-stratification and post-stratification contexts. In Section 5, the results of simulation studies to assess the type I error for each approach are described. The results of applying the proposed method to data from previous randomized clinical trials are described in Section 6. Finally, concluding remarks are given in Section 7.

2 Notation and traditional approaches

3 Proposed test under pre-stratification

In this section and Section 4, we introduce a new statistic for testing H0:δ=0 based on eq. (2) under the assumption that the number of patients in each stratum follows multinomial distributions:

In a pre-stratification analysis, the patients are randomized within each stratum. Therefore, we can assume that the ratio of the number of patients in the two groups is exactly nTi/nCi=kT/kC in arbitrary stratum i, where kT and kC represent positive integers. That is, nCi can be represented as

for i=1,2,⋯,K.

The variance of the estimator of the overall treatment effect δˆ is represented as

By tedious calculation, the exact formulation of this variance is derived as

The detailed derivation of eq. (8) is given in Appendix A. With replacement of the unknown parameters pTi and pCi by pˆi in eq. (5), replacement of δi and δ by the estimator, and under the null hypothesis, a new test statistic is proposed as follows:

When πi is unknown, its value is replaced by πˆi in eq. (1). This test statistic tests the significance of the overall treatment effect as χC2 by using the chi square distribution with 1 degree of freedom.

In the case of pre-stratification, by using eq. (6), the estimator of the variance in eq. (4) can be reduced to

On the other hand, if πi is replaced by πˆi, the estimator of the variance in eq. (8) can be represented as

Because the second term in eq. (10) is always a positive value, if the number of patients is not fixed at the beginning of a trial, and randomization is conducted by pre-stratification, the value of the statistic in eq. (2) will be larger than that in eq. (9). As a result, using the Cochran test in such situations carries the risk of over-detecting the treatment effect. In addition, the value of this term approaches 0 when the true value of δi equals δ in the arbitrary stratum because δˆi and δˆ are consistent estimators of δi and δ, respectively. In this case, the value of the statistic in eq. (9) will be close to that in eq. (2) because of the increase in the sample size. When the sample size is small, because the variability of the difference between δˆi and δˆ in each stratum is large, the second term in eq. (10) tends to have a large value, and as a result, our proposed statistic will yield a more conservative result.

4 Proposed test under post-stratification

In a post-stratification analysis, the randomization is conducted for all patients. Although the total number of patients that will be assigned to either of the two groups is fixed, the ratios of the number of patients in the strata could differ according to each stratum. Therefore, in this case, we can only assume that NT and NC are fixed. Of course, in this case, the variation in the number of patients is expected to be larger than that in the case of pre-stratification, because nTi and nCi follow independent distributions.

The variance of δˆ can be calculated from eq. (7) as in the pre-stratification case. However, as an additional difficulty that is distinct from the previous case, it is necessary to calculate the expected value and variance of a complex function that is constructed from nT1,nT2,⋯,nTK and nC1,nC2,⋯,nCK. Although the details of the calculation are given in Appendix B, to address this problem, we used the multivariate first-order Taylor expansion around the expected values. As a result, the approximation of the variance of δˆ is given by

With replacement of unknown parameters pTi and pCi by pˆi in eq. (5), and replacement of δi by the estimator, the estimation of eq. (11) is given by

Then, we propose a new test statistic based on eq. (2) as follows:

As in the case of χpre2, πi is replaced by πˆi in eq. (1) when πi is unknown. The test statistic is compared to the chi square distribution with 1 degree of freedom to test the significance.

In post-stratification, the difference between VarˆC(δˆ) and Varˆpost(δˆ) cannot be as easily compared as in the pre-stratification case with eq. (10). If πi is unknown, the difference between the first term of eq. (12) and VarˆC(δˆ) is given as

Therefore, the difference between these two values can be considered by the magnitude of the difference between the heterogeneities of the two groups, which are calculated from the total number of patients (NTNC/N2) and the number of patients in each stratum (nTinCi/ni2). When the total number of patients in the two groups is completely balanced (NT=NC), the value of eq. (14) always becomes positive, because the value of nipˆi(1−pˆi) is at least 0. In this case, if the numbers of patients in each stratum are completely balanced (nTi=nCi, ∀i), then eq. (14) becomes 0. When the total number of patients becomes large, the number of patients in each stratum becomes balanced, and as a result, the value of eq. (13) approximates eq. (2). On the other hand, if the total number of patients is small, the imbalance of patients in each stratum becomes large within the poststratification framework, and as a result, the value of eq. (13) becomes more conservative than eq. (2).

Even if the total number of patients in the two groups is unbalanced, the value is expected to become positive since the number of patients in each stratum will be unbalanced in many cases. As a special case, when the total number of patients is unbalanced and the true values of pTi, pCi, and πi are extremely skewed, eq. (14) could become a negative value. In almost all cases, however, the first term of eq. (12) becomes greater than VarˆC(δˆ). In addition, the second term of eq. (12) always becomes greater than 0. As a result, the value of the test statistic based on eq. (13) will be less than χC2 in many situations. In addition, for the case of this unbalanced total number of patients, when the total number of patients becomes large, the difference between eqs (2) and (13) is expected to become small because the value in the first pair of parentheses of eq. (14) approaches 0.

5 Simulations

We here present the results of simulation studies to demonstrate the excessively high type I error obtained with the Cochran test and the reasonably controlled type I error obtained using the proposed test under the situation of a stratified randomized clinical trial. As mentioned above, there are several tests in addition to the Cochran test for assessing the overall treatment effects. However, since our proposed approaches are constructed based on eqs (2) and (3), we focus only on the comparison to the Cochran test in this study. To assess the type I error, we generated the simulated data from hypothetical stratified trials under several situations of H0:δ=0 and compared the percentage of times the null hypothesis was rejected among all tests according to the significance level α=0.05.

For each setting, the simulations were repeated 50,000 times. The number of treatment patients in strata (nT1,nT2,⋯,nTK) is given by a multinomial random number with NT and (π1,π2⋯,πK). The number of patients in the control group is given by (nC1,nC2,⋯,nCK)=(nT1,nT2,⋯,nTK)×kC/kT for the pre-stratification case. In the post-stratification case, (nC1,nC2,⋯,nCK) is given by a multinomial random number with NC and (π1,π2⋯,πK).

For both pre-stratification and post-stratification, the settings of the simulations were equivalent. The only difference was whether or not there is the exact setting of the ratio of the number of patients in the two groups kT/kC. We set kT/kC=1 for all simulations in the pre-stratification case. In the post-stratification case, we set NT=NC for all settings. Therefore, the total number of patients in the two groups was set to be equivalent in all simulations for both cases. The number of strata K was set to 2, 5, or 10. The setting of the low number of strata is assumed for a situation in which there is a general covariate such as sex or stratified age that should be considered to affect the associations between the two groups and the response in a randomized clinical trial. On the other hand, the setting with a high number of strata is assumed to represent a large-scale study such as a multicenter trial or meta-analysis.

In the case of K=2, the success probabilities of the two groups were set to (pT1,pT2)=(pC1,pC2)=(0.7,0.4). In the cases of K=5 and K=10, the probabilities were set to (pT1,⋯,pT5)=(pC1,⋯,pC5)=(0.7,0.6,0.5,0.4,0.3) and (pT1,⋯,pT10)=(pC1,⋯,pC10)=(0.75,0.7,0.65,0.6,0.55,0.5,0.45,0.4,0.35,0.3), respectively. The other settings and results are shown in Table 1-Table 4. In all Tables, the percentages of times the null hypothesis was rejected by the Cochran test (χC2) and the proposed tests (χpre2 or χpost2) are shown for the cases when (π1,π2,⋯,πK) is known and unknown.

Table 1 shows the results for the pre-stratification and K=2 case. As expected, the percentage of times the null hypothesis was rejected was lower with the proposed approach than with the Cochran approach in all cases. In the two-strata case, the Cochran test showed a nearly nominal significance level in almost all cases. When the sample size was small, however, the percentage of times H0 was rejected became too high, regardless of the strata proportion in the population. In addition, the proportion of times H0 was rejected increased when the heterogeneity of the strata became very high. However, in such situations, the proposed test could also control the nominal significance level well. During a comparison of the results of the proposed tests when (π1,π2) was known and unknown, the test that estimates (π1,π2) tended to become more conservative when the sample size was small. After the number of samples was increased, these two test statistics quickly showed convergent values.

Table 2 shows the results for the pre-stratification and K=5 and 10 cases. If the number of strata was large, the Cochran test tended to over reject H0 in almost all situations. On the other hand, the proposed test tended to be more conservative, especially when the sample size was small. Both tests approximated to the nominal level of the test as the number of samples increased. As the reason for this finding, the effect of the second term in eq. (10) can be considered. That is, when the total number of patients is small, the variation between estimators of the treatment effect in each stratum becomes large, and the value of the second term in eq. (10) tends to become large. Consequently, our proposed test tends to be conservative in this case. This second term in eq. (10) approximates 0 if the number of patients is increased in this simulation setting, and the result of our proposed test approximates the result of the Cochran test. Judging by the results in Table 1 and Table 2, if the sampling structure in a stratified randomized trial is a prestratification case, we recommend our approach to control the type I error at the nominal level. Especially, when the number of patients included in a stratum is less than 25, we strongly recommend our approach.

Table 3 and Table 4 show the results for the case of post-stratification with the same settings shown in Table 1 and Table 2, respectively. Essentially, the results were the same as observed in the corresponding pre-stratification cases. However, in the post-stratification analyses, the tendency of over rejection of the null hypothesis by the Cochran test was stronger. The proposed approach showed better performance than the traditional approach in all situations, especially for the results shown in Table 3. In accordance with these results, we recommend the proposed approach when the sampling structure is the poststratification framework. In particular, when the number of patients included in a stratum is less than 25, we strongly recommend our approach as in the case of the prestratification.

These simulation studies confirmed that the Cochran approach has a risk of obtaining a higher type I error than the nominal level in both pre-stratification and post-stratification situations. In particular, when the number of patients is small in a post-stratification analysis, this risk becomes higher. On the other hand, as expected, our proposed approach could control the type I error well. Although the proposed approach tends to become slightly conservative, it can be considered to more appropriate than the Cochran test with respect to consideration of the significance of a type I error in clinical trials. Of course as the trade-off of increasing the estimation value of the variance of δˆ, our approach has lower power than the traditional approach. However, as previously mentioned, the type I error has to be controlled by the nominal significance level first in almost all situations.

6 Examples

We here present two examples of the analysis of randomized clinical trial data using the proposed methods.

7 Conclusion

In this paper, we focused on the variability in the number of patients in different strata of randomized clinical trials with a binary endpoint. In traditional approaches to test the significance of the overall treatment effect, the number of patients in each stratum is assumed to be fixed. However, in several situations of comparative trials, the patient number in different strata is allowed to vary. Therefore, using traditional approaches in such situations comes with a potential risk of underestimating the variability of the estimated treatment effect, which consequently increases the type I error over the nominal level. This problem can be illustrated with the results of simulation studies.

To deal with this problem, we calculated the new variance of the estimated treatment effect defined by the risk difference in both pre-stratification and post-stratification situations. We assumed that the number of patients follows a multinomial distribution. As seen in eqs (10), (14), and the results of the simulation studies, the proposed approach could effectively control the type I error in almost all situations. Furthermore, in examples of patient data, our approaches clearly resulted in greater p-values than those yielded by the Cochran test.

Although we focused on the test of the significance of the overall treatment effect in this study, our approach can easily be extended to the calculation of the confidence interval of the treatment effect. Of course, in such cases, the interval constructed from our approach would be expected to include the true overall treatment effect with a nominal level, although the length of the interval would become wider than that of traditional approaches.

From the perspective of increasing the variance, our approach can be considered to be similar to an approach that considers a random effect of the treatment between strata. However, the assumptions of the two approaches are essentially different. In fact, the rejection probabilities were almost 0 for all situations modeled in our simulation when using DerSimonian’s approach [13], which is a typical method used in meta-analyses to consider random effects with a binary endpoint.

Finally, our approach can be extended to applications with many methods, including other criteria for evaluating a treatment effect such as relative risk or odds ratio, other sampling structures such as a case-control study, and other endpoints such as a continuous variable or inclusion of censored cases. In several cases, it is difficult to calculate the variances exactly, and the obtained formula would be complex. However, it is worth carefully considering the variation in each of these situations from the point of view of aiming to achieve precise control over the type I error.