A Comparison of Some Approximate Confidence Intervals for a Single Proportion for Clustered Binary Outcome Data

1 Introduction

Binary outcome data sampled from clusters arise frequently in many biomedical, toxicological, clinical medicine, and epidemiological applications. The observed binary outcome data often exhibit greater or lesser variability than that predicted by a simple binomial model, referred to as over/under-dispersion [1]. There are several reasons that may lead to the over/under-dispersion in binary data. For instance, in a case-controlled study of familial aggregation of chronic obstructive pulmonary disease (COPD) [2], siblings within each family are correlated and the number of impaired pulmonary function (IPF) cases per family may be over-dispersed as compared to a binomial model. The 95% profile based confidence interval for intraclass correlation proposed by Saha ([3]) using the data in the study mentioned above (see Table 5 of Liang et al. [2]) is (0.0593, 0.4006), which supports the significance of within-family correlation. As a result, it shows that the observed variance 0.1418 in the estimated proportion of the IPF cases per family is 1.38 times larger than the predicted variance 0.1026 obtained using a binomial model. This concludes that standard approaches (see, for example, Hogg and Tanis [4], pp. 308–310) of analyzing such data that ignore the cluster structure may result in underestimation of the true standard error of the estimated infected rate when the correlation between siblings per family is positive. Furthermore, inference methods concerning the parameters of interest based on the binomial model in such data may significantly inflate the Type I error rate [5]. Although a number of confidence intervals for a single proportion based on clustered data have been studied for complex survey data, little attention has been paid to model-based approaches for inferring about the proportion in the analysis of clustered binary data. Kleinman [6] studied the properties of the maximum likelihood (ML) and the method of moments (MM) estimators for the proportion parameter based on parametric and semiparametric models for clustered binary data. Based on several model structures, Paul and Islam [7] investigated the joint estimation of the proportion and dispersion parameters in terms of bias and efficiency. Surprisingly, these approaches were not extended to investigate the coverage probabilities of confidence interval estimation of the proportion.

Developing a confidence interval for the proportion parameter using clustered binary data is an important problem. For example, in the diagnostic accuracy of contrast-enhanced multi-detector row spiral computed tomography coronary angiography [8], interval estimates of sensitivity (true positive rate) and specificity (true negative rate) are often used at the patient level, at the coronary artery level, and at the coronary artery segment level. Another example involves the estimation of sensitivity and specificity to assess the accuracy of radiologists’ readings in a mammogram screening study [9], where the proportion of positive readings in cancer cases and the proportion of negative readings in non-cancer cases within each radiologist may be over-dispersed. To make inferences on sensitivity and specificity, one usually uses the inference methodology developed for a single proportion. There is an abundance of literature pertaining to inferences for a single proportion based on non-clustered binary data (for example, Agresti and Coull [10] and Newcombe [11]. However, little attention has been paid to such inferences in the case of small to moderate sample size clustered binary data using parametric and semiparametric models. For complex survey data, some authors have developed alternative methods extending those derived for non-clustered binary data. In particular, the modified Clopper-Pearson (MCP) method and the modified Wilson score (MWS) method are recommended for analyzing complex survey data (see Korn and Graubard [12], page 65). However, in some situations the MCP method is somewhat conservative while the MWS method shows lack of coverage when the variability of the weights is small or large. Moreover, the sampling weights are not readily available in biomedical applications though they are required to calculate the effective sample size in order to find these intervals.

Lui [13] derived three methods based on model assumptions using the estimation of the intraclass correlation by analysis of variance, but the performances of these methods were not examined. Rutter [14] introduced bootstrap interval methods for sensitivity and specificity to measure the diagnostic accuracy with patient-clustered data using bootstrapping to estimate the variance, but these methods are computationally demanding. For the balanced data set-up, Kim and Lee [9] proposed an asymptotic confidence interval for a single proportion based on the beta-binomial distribution which works well for larger proportions when the cluster sizes are over 25, but suffers from serious under-coverage for small numbers of clusters.In many practical problems, the cluster sizes are often not equal and small (\lt15) or the number of clusters is small to moderate (see, for example, Zhou et al. [8], page 112). In order to assess the accuracy of computer-aided detection enhanced computed tomography colonography for the detection of polyps, Zhou et al. [8] obtained the asymptotic confidence interval for sensitivity of clustered binary data using a ratio estimator for the variance given by Rao and Scott [5]. However, this method shows serious under-coverage (see, Figure 1 and Paul and Zaihra [15], p. 4219).

Figure 1:

The observed coverage probability and the expected interval length of 95% nominal confidence intervals for the proportion π based on the 15 methods discussed in Section 3. Each box plot was constructed based on 180 parameter combinations.

The main focus of this paper is to develop asymptotic confidence intervals for a single proportion arising in cluster studies. In particular, in Section 3 we propose two new approaches based on the Wilson score and profile likelihood that will properly incorporate the intraclass correlation structure. In addition, we consider a number of extensions of existing methods in order for them to be feasible for clustered binary outcome data. Section 4 conducts a simulation study to assess the performances of these intervals in comparison with two existing methods recommended for analyzing survey data in terms of coverage and interval length. The methods developed in this paper are applied to analyze medical data sets in Section 5. Some concluding remarks are given in Section 6.

2 Models

3 Confidence intervals for the single proportion

Since we will deal with and compare a large number of methods for constructing confidence intervals (CI), we now list all of them and their acronyms for easy reference:

4 Simulation studies

The primary goal of our simulations was to provide guidance in the selection of an appropriate confidence interval for the proportion, based on clustered binary data, by assessing the performance of the 15 intervals considered in this paper, in terms of the observed coverage probability and the average interval length using the pre-assigned confidence levels of 90% and 95%. In some cases, there are only a few clusters with unequal and variable cluster sizes. For example, in multicenter clinical trials or studies that validate assay sensitivity and specificity across several labs there may only be a few clusters (few clinical centers, few labs) with large cluster sizes. In some other applications, e.g., in ecology and parasitology, the clusters could be single animals with very few repeated observations available, i.e., cluster sizes are very small [16]. Based on the above realizations, we considered six different configurations of cluster sizes with different numbers of clusters: (i) fixed cluster sizes (ni: 12, 7, 6, 6, 7, 8, 10, 7, 8, 6, 11, 7, 8, 9, 2, 7, 9, 7, 11, 10, 4, 8, 10, 12, 8, 7, 1) of the control group (k= 27) and fixed cluster sizes (ni: 5, 11, 7, 9, 12, 8, 6, 7, 6, 4, 6, 9, 6, 7, 5, 9, 1, 6, 9) of the low dose group (k = 19) as in Table 2 of Rao and Scott [5]; (ii) fixed cluster sizes (ni: 1, 3, 5, 5, 6, 7, 7, 8, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 16, 17, 17) of the control group (k = 73) as in Table 3 of Bowman and George [24]; and (iii) variable cluster sizes generated from the empirical distribution (ED) of 523 litter sizes where the litter sizes range from 1 to 19 with a mean of 12 and standard deviation 2.98 (see, Figure W1 in the Supplementary Materials for further details), quoted by Kupper et al. [25], for k= 20, 50, 100. Ten values of π = 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, and three values of ϕ = 0.1, 0.3, 0.5, were considered in our investigation. However, we only present representative results for brevity. Following Paul and Islam [17], 10,000 data sets were generated from the beta-binomial distribution. Following Paul and Saha [22], we also considered negative estimates of ϕ in our simulation, with the restriction that ϕˆ>−1/(nmax−1), where nmax = max{ni, i=1,…,k}. The observed coverage probability (CP) and the expected interval length (EL) for two-sided confidence intervals (lt, ut) for π were obtained by

where I=1 if lt≤π≤ut, and I = 0, otherwise.

The coverage properties and expected lengths for all the methods are almost identical for both the 90% and 95% pre-assigned confidence levels. We present simulation results for only a pre-assigned confidence level of 95% with the results for a 90% nominal confidence level being provided in the Supplementary Materials (see, Figures W2–W6 and Tables W5–W8). The box plots and the medians of the observed CPs and the ELs obtained from the 15 interval procedures for the 180 (6 values of k, 10 values of π, and 3 values of ϕ) parameter combinations are reported in Figure 1 and Table 1, respectively. The horizontal lines in the coverage probability figure indicate the CPs 0.94, 0.95, and 0.96, respectively. From Figure 1 and Table 1, it can be seen that the median CPs for all methods are, in general, close to the pre-assigned confidence level, except for EQL, DEQL, G₁, and MCP. The EQL, DEQL, and MCP intervals are conservative as their median CPs lie above 96%, while the G₁ intervals are liberal as evidenced by median CPs below 94%. These results are clearly seen in Figure 1 where almost 75% of the CPs for EQL, DEQL, and MCP lie above the horizontal line at 0.96, and a similar proportion of the CPs for G₁ lie below the horizontal line at 0.94. Although G₁ and R₁ have liberal behaviors in coverage (also reported by Paul and Zaihra [15], p. 4219), their adjusted versions, G₂ and R₂, clearly show much improvement in coverage as most of their CPs lie between the horizontal lines at 0.94 and 0.96. The remaining eight procedures: ML, QEE, WA₁, WA₂, WI₁, WI₂, PL, and MWS, perform reasonably well. However, in some situations, ML, QEE, WA₁, and WA₂ tend to be liberal (i.e., CPs < 0.94). Based on coverage, overall the WI₁, WI₂, PL, and MWS intervals outperform the other intervals. As expected, the ELs for EQL, DEQL, and MCP are wider than those for the other intervals considered here. In general, the ELs for ML and PL are shortest, while the ELs for the remaining nine intervals are very similar except for QEE. However, WI₂ and WA₂ have a slight edge in EL compared to WI₁ and WA₁.

We now evaluate the methods stratified by different parameter space points. It has been seen from the above results that EQL, DEQL, and MCP are somewhat conservative, and the CPs for G₂ and R₂ are better than those for G₁ and R₁. However, it shows that MCP has a less conservative property than those of EQL and DEQL. In the interest of brevity, we present the observed CP and EL results only for the 11 generally more competitive methods, namely, PL, WI₁, WI₂, WA₁, WA₂, R₂, G₂, ML, QEE, MCP, and MWS. Since the results for fixed cluster sizes between k=19 and k=27 as well as for ED cluster sizes between k=20 and k=30 are almost identical in most data situations, in Figures 2–5 we present the observed CPs and the ELs only for fixed cluster sizes of k = 19 and 73 and for ED cluster sizes of k = 20 and 50. In addition to Figures 2–5, the simulation results for selected parameter combinations are also presented in Tables 2 and 3 to clearly show how the results of CPs and ELs are affected by different subsets of π and cluster sizes.

Figure 2:

The observed coverage probability of 95% nominal confidence intervals for the proportion π based on the 11 methods for fixed litter sizes. Each box plot was constructed based on 10 parameter combinations.

Figure 3:

The observed coverage probability of 95% nominal confidence intervals for the proportion π based on the 11 methods for ED litter sizes. Each box plot was constructed based on 10 parameter combinations.

Figure 4:

The expected interval length of 95% nominal confidence intervals for the proportion π based on the 11 methods for fixed litter sizes. Each box plot was constructed based on 10 parameter combinations.

Figure 5:

The expected interval length of 95% nominal confidence intervals for the proportion π based on the 11 methods for ED litter sizes. Each box plot was constructed based on 10 parameter combinations.

The results in Figures 2 and 3 and Table 2 show that, irrespective of fixed or variable cluster sizes, the coverage properties of all the methods are very similar. As expected, the variations of CPs for different values of π for all the methods decrease as k increases (see, for example, Figure 2(iii) for k=19 and Figure 2(iv) for k=73). Although the median coverage probability for ML is between 94% and 95%, we see inconsistent coverage in some situations. For instance, when π=0.05, the CPs are much larger than the nominal level (see, for example, Table 2 when k=19 and k = 20), and for moderate π’s, the CPs are much smaller than the nominal level (see, for example, Table 2 when k=19, ϕ=0.5, and 0.15≤π≤0.35). Like the ML intervals, the QEE method shows inconsistency in coverage when π<0.15. However, in most data situations the QEE CPs are between 94% and 95% (see, for example, Table 2 when k=73). R₂, G₂, WA₁, and WA₂ have reasonable coverage properties, but for small values of π and k, these methods tend to be very conservative, whereas WA₁ and WA₂ tend to be very liberal in a few cases (see, for example, Table 2 for ϕ=0.5 and 0.25≤π≤0.35 when k=20). For small values of k, the CPs for MCP are larger than 96% for almost all situations, while the CPs for MCP are between 95% and 96% for larger values of k. Overall, the WI₁, WI₂, PL, and MWS intervals have the best coverage properties, meaning the CPs of these methods are very close to the nominal level in almost all the data situations considered here. However, in most situations, WI₁, WI₂, and MWS show better consistency in coverage than PL.

As expected, we see from the results in Figures 4 and 5 and Table 3 that the ELs for all the methods decrease as k increases. In addition, the ELs for all the methods increase when the risk rate, as well as the deviation from independence among observations within the same cluster, increase. For instance, ranges for the ELs in Figure 4(v) for ϕ=0.5 are larger than those in Figure 4(v) for ϕ=0.1, and the ELs for all the methods when π=0.05 are much larger than those when π=0.45 (see, Table 3). It should be noted that there are no significant differences in the ELs between the fixed and variable cluster sizes. In general, ML and PL provide the lowest ELs, while QEE and MCP show the largest ELs, particularly for small values of k. Although the ELs for ML are among the smallest, in some situations this is at the expense of serious under/over-coverage. The remaining seven methods have very similar ELs throughout the parameter space points considered here except for MWS whose ELs were found to be slightly higher than the ELs of the other six methods. Further details regarding the above results can be found in the Supplementary Materials (see, Tables W1–W4).

5 Applications to real data analysis

We apply all 15 CI methods considered in Section 3 to three data sets: (i) CTC images data [8], (ii) chronic obstructive pulmonary disease (COPD) data [2], and (iii) screening mammogram data [9]. The number of clusters and cluster sizes varied among these studies.

6 Discussion and concluding remarks

This paper considers 13 asymptotic CIs for binary outcome data, taken from clusters, assuming a beta-binomial distribution and semi-parametric models such that only the first two moments of the responses need be specified. The results of our simulation studies in Section 4 suggest that the two versions of the Wilson score, the modified Wilson score, and the profile likelihood methods are preferable as their observed CPs are very close to the nominal coverage level. However, the Wilson score and the modified Wilson score methods are preferred to the profile likelihood method in that they are well controlled around the desired coverage level, though the profile likelihood method is preferred to the two versions of the Wilson score and the modified Wilson score in the sense that it generally possesses shorter ELs in almost all data situations.

Our results in this paper depend on the assumption of the beta-binomial distribution and semi-parametric models specified by only the first two moments of the response variable (which we considered similar to the mean and variance of the beta-binomial distribution). Moreover, we considered a common correlation structure assumption among the observations within the same cluster which is often applicable to family studies; however, one can extend this research by using a more general correlation structure arising in many genetic epidemiology studies.

In our simulation studies it appeared that in some situations the asymptotic confidence intervals based on ML, EQL, DEQL, and QEE showed serious lack of coverage, particularly EQL and DEQL. These CI procedures rely largely on the asymptotic normality distribution assumption for the estimate of the parameter, which may not hold in some situations, especially for small sample sizes or small parameter values. Moreover, the problem of lack of coverage could arise due to estimation of the asymptotic standard error. Further research that can solve these issues by incorporating alternative distributional approximations, such as parametric, nonparametric, and double bootstrap, to yield greater coverage accuracy would clearly be worthwhile.