Nonparametric methods of statistical inference for double-censored data with applications

1 Introduction

This article introduces new nonparametric statistical methods for prediction using the past data that contain right-censored observations and left-censored observations simultaneously, where these kind of data are called double-censored data in the literature. The methods are proposed to learn about one future observation based on the original sample with few mathematical assumptions. For real-valued data, Hill [1,2] presented the A ( n ) assumption for prediction when there is few knowledge about the underlying distribution, and this assumption provides certain probabilities for one future observation based on the past observations. For right-censored data, Berliner and Hill [3,4] generalized the A ( n ) assumption, and the generalizations are considered for survival analysis. In this article, regarding methods are developed by using the information of censoring observations, and they are referred to by generalization type A and B of A ( n ) assumption.

This article is organized as follows. In Section 2, a brief overview of the A ( n ) assumption is given for real-valued data. Section 3 presents the generalizations of A ( n ) assumption for right-censored data. Section 4 presents the generalizations of A ( n ) assumption for double-censored data along with their corresponding justifications. In Section 5, we present bounds for probabilities and survival functions based on the generalizations of A ( n ) assumption proposed for double-censored data. In Section 6, the new proposed generalizations are used with two applications. In this article, we assume that ties can occur with probability zero to make notation simple, but Section 7 briefly discusses the way that ties can be dealt with. The final section provides some concluding remarks and discusses some future-related research.

2 A ( n ) assumption for real-valued univariate data

In 1968, Hill [1] introduced the A ( n ) assumption for prediction when there is no prior knowledge about the underlying distribution. The data support is partitioned into n + 1 intervals using the observed data points and a probability 1 n + 1 is assigned to each created interval. Let X 1 , X 2 , … , X n be continuous and exchangeable real-valued random quantities, and let x 1 , x 2 , … , x n be the corresponding observed data points. Furthermore, let x ( 1 ) < x ( 2 ) < … < x ( n ) be the ordered observations and let x ( 0 ) = − ∞ (or x ( 0 ) = 0 for positive random quantities) and x ( n + 1 ) = ∞ . For one future observation X n + 1 , the A ( n ) assumption is

where i = 0 , 1 , 2 , … , n .

It should be noted that the A ( n ) assumption is suitable to provide bounds for probabilities, known as imprecise probabilities, by using the theorem of probability proposed by De Finetti [5], but it is not good to derive precise probabilities for many functions of interest. The bounds created for probabilities can lead to valuable information in the case of uncertainty of event or in the case of indeterminacy caused by restricted information [6,7]. Augustin and Coolen [8] derived strong consistency properties for nonparametric predictive inference in interval probability theory based on the A ( n ) assumption, and multiple examples have been presented by Coolen [9], Coolen and Coolen-Schrijner [10], Coolen et al. [11], Coolen and Van der Laan [12].

Based on data including n event observations, the A ( n ) assumption provides a partially specified predictive probability distribution for one future observation X n + 1 via the probabilities assigned to the intervals ( x ( i ) , x ( i + 1 ) ) , where i = 1 , 2 , … , n , with x ( 0 ) = 0 and x ( n + 1 ) = ∞ . These probabilities can lead to derive lower and upper probabilities for any event of interest in terms of X n + 1 [8]. If we are interested in the event X n + 1 ∈ A , with A a set of the nonnegative real values, then the lower probability for this event, referred to by P ̲ ( X n + 1 ∈ A ) , is derived by summing only the probabilities for X n + 1 on intervals ( x ( i ) , x ( i + 1 ) ) , which are completely within the set A . The upper probability for the event X n + 1 ∈ A , referred to by P ¯ ( X n + 1 ∈ A ) , is derived by summing all the probabilities for X n + 1 on intervals ( x ( i ) , x ( i + 1 ) ) , which have a nonempty intersection with the set A .

The A ( n ) assumption has been discussed for multiple statistical inferences in the literature, see, e.g., Hill [2] and De Finetti [5] for more detailed presentation, and the assumption has been contributed with multiple important related works, see, e.g., Dempster [13] and Lane and Sudderth [14] for detailed information. For right-censored data, Mark Berliner and Hill [3] and Coolen and Yan [4] proposed two versions of A ( n ) , and the proposed assumptions will be presented in the next section.

3 Generalizations of A ( n ) assumption for right-censored univariate data

The A ( n ) assumption for real-valued data is introduced as certain predictive probabilities are assigned to open intervals created by the observed data points with no further assumptions or restrictions on the spread of probabilities within the intervals. In 1988, the A ( n ) assumption is generalized for data containing right-censored observations by Mark Berliner and Hill [3]. They use the same technique and provide a partial probability distribution via certain values for next future observation X n + 1 , and the probability masses are assigned to open intervals with no more constraints on the spread of the probability mass within each interval. Let X 1 , X 2 , … , X n be exchangeable positive random quantities, and there are u event observations, and v right-censored observations. The event and right-censored observations are ordered as 0 < t ( 1 ) < t ( 2 ) < … < t ( u ) , where 0 ≤ u ≤ n , and 0 < c ( 1 ) < c ( 2 ) < … < c ( v ) , where v = n − u , respectively. They assign a certain probability for one future observation X n + 1 to be fallen in any two ordered event observations ( t ( u ) , t ( u + 1 ) ) by the following formula:

where λ ( u ) = 1 n + 1 − u − C ( u ) , C ( u ) = # { censors < t ( u + 1 ) } and P ( 0 ) = λ ( 0 ) .

This proposed assumption was the first attempt to generalize A ( n ) assumption for right-censored data, and it was a good start to deal with the case of data including right-censored observations. Coolen and Yan [4] provided an alternative approach, named by the right-censoring A ( n ) assumption, which is nicer because it does not neglect the censored observations as Berliner and Hill does to create the intervals partitioned the sample space.

The right-censoring A ( n ) assumption, rc - A ( n ) , provides a partial probability distribution for one future observation, and it is specified via M-function values [4]. The random quantities X 1 , X 2 , … , X n are assumed to be exchangeable, nonnegative and real-valued, and there are u event observations, and v right-censoring observations. The event and right-censoring observations are ordered as 0 < t ( 1 ) < t ( 2 ) < … < t ( u ) , where 0 ≤ u ≤ n , and 0 < c ( 1 ) < c ( 2 ) < … < c ( v ) , where v = n − u , respectively. Let the interval I i = ( t ( i ) , t ( i + 1 ) ) , for 0 ≤ i ≤ u , they ordered the right-censoring times in each interval I i by c 1 i < c 2 i < … < c l i i , where l i is the number of censors in I i . Finally, they formed the open intervals as ( t ( i ) , t ( i + 1 ) ) and ( c k i , t ( i + 1 ) ) , where 1 ≤ k ≤ l i .

From studying the rc - A ( n ) assumption, two interesting notes have been experienced, and it is good to list them to have a wider picture for the assumption. First, Coolen and Yan [4] presented a new version of the A ( n ) assumption for data containing right-censored observations by using the same technique of A ( n ) assumption, but the intervals are overlapped due to the censored observations. They provided a partial probability distribution via the M-function values for a random quantity X n + 1 , and the probability masses are assigned to open intervals with no more constraints on the spread of the probability mass within each interval. The probability mass specified to such interval ( a , b ) is referred to by M X n + 1 ( a , b ) and interpreted by M-function value for X n + 1 on ( a , b ) . Second, the M-function values are limited between 0 and 1, and the values have to sum up to one over all intervals created [4].

4 Generalizations of A ( n ) assumption for double-censored univariate data

This section provides two generalizations of A ( n ) assumption along with their justifications. For the justifications, we will include a detailed information of the exact nature of the noninformative censoring assumption implicit in the generalizations, and the justifications will be provided in two stages. First, the influence of double-censored observations on the A ( n ) assumption will be presented, with no further assumptions added. In the second stage, further postdata and predata assumptions, which are strongly related to A ( n ) , are considered to derive partially specified predictive probability distributions for the random quantities that were double censored. The further postdata assumption will be for right-censored data, and the further predata assumption will be for left-censored data.

Based on the A ( n ) assumption, a partially specified probability distribution for X n + 1 is provided by using the observed data points. As first stage in our justifications of the generalizations of A ( n ) , the influence of double-censored data on the partially specified probability distribution for X n + 1 is considered without further constraints, and this leads to generalize A ( n ) assumption for double-censored data. This generalization will be referred to by A ˜ ( n ) , and the definition is justified as follows:

The justification of A ˜ ( n ) assumption, in relation to the A ( n ) assumption, is as follows. The intervals in the form of ( t ( i ) , t ( i + 1 ) ) are each assigned a minimum probability mass of 1 n + 1 , by the A ( n ) assumption. If we consider one such an interval, then the total mass in it could be more than 1 n + 1 due to the presence of double-censored observations. Any additional probability mass due to such double-censored observations does not have to be restricted to lie within this interval, without any assumptions, leading us to the probabilities on intervals from a right-censored observation to infinity and from 0 to a left-censored observation. For the case of having a right-censored observation in ( t ( i ) , t ( i + 1 ) ) , then the unobserved data point t r c corresponding to that right-censored observation could occur in ( t ( i ) , t ( i + 1 ) ) . This leads to assign a probability 1 n + 1 to the interval ( t ( i ) , t r c ) and another probability 1 n + 1 to the interval ( t r c , t ( i + 1 ) ) . For the case of having a left-censored observation in ( t ( i ) , t ( i + 1 ) ) , then the unobserved data point t l c corresponding to that left-censored observation could occur in ( t ( i ) , t ( i + 1 ) ) . This leads to assign a probability 1 n + 1 to the interval ( t ( i ) , t l c ) and another probability 1 n + 1 to the interval ( t l c , t ( i + 1 ) ) . However, due to the lack of information about the true times of double-censored observations, we cannot assign a probability 1 n + 1 to the subintervals. Therefore, the probability for X n + 1 to be fallen in ( t ( i ) , t ( i + 1 ) ) is set equal to 1 n + 1 .

The only information known about a right-censored observation r c ( j ) is that the corresponding event would occur after time r c ( j ) . This implies that if this time were observed, then one of the intervals in the form of ( t ( i ) , t ( i + 1 ) ) would be split into two intervals, and each interval has a probability 1 n + 1 based on the A ( n ) assumption. However, because it is only known that the corresponding event would exceed r c ( j ) , the only statement about this probability mass 1 n + 1 for X n + 1 that can be justified, with no more constraints, is that it will fall in ( r c ( j ) , ∞ ) , and hence, P ( X n + 1 ∈ ( r c ( j ) , ∞ ) ) = 1 n + 1 .

For a left-censored time l c ( w ) , we know that the corresponding event occurred before time l c ( w ) . From this information, the event is occurred in one of the intervals ( t ( i ) , t ( i + 1 ) ) , and this leads to split the interval ( t ( i ) , t ( i + 1 ) ) to two intervals, and each one has a probability 1 n + 1 based on the A ( n ) assumption. But, because it is only known that the corresponding event occurred before l c ( w ) , the only statement about this probability mass 1 n + 1 for X n + 1 that can be justified, with no more constraints, is that it will fall in ( 0 , l c ( w ) ) , and hence, P ( X n + 1 ∈ ( 0 , l c ( w ) ) ) = 1 n + 1 .

5 Imprecise probabilities based on the generalizations of A ( n ) assumption

The use of the partially specified predictive probability distribution for the next future observation X n + 1 is introduced in this section, through the probabilities as given by the generalizations of A ( n ) assumption. Such inferences have a predictive nature in terms of the next future observation X n + 1 along similar lines as nonparametric predictive inferences based on the A ( n ) assumption for real-valued data [8–12] and based on the right-censoring A ( n ) assumption for right-censored data [15–19]. For many events of interest regarding to X n + 1 , the probability values proposed by equations (6) and (7) allow bounds for probabilities to be derived analogously as for A ( n ) -based inference [8], where the maximum lower bound is referred to by the “lower probability,” and the minimum upper bound is referred to by the “upper probability.” These bounds follow the terminology proposed in the theory of imprecise probabilities [6,7].

Based on data including u event observations, v right-censored observations, and k = n − ( u + v ) left-censored observations, each generalization of A ( n ) assumption provides a partially specified predictive probability distribution for one future observation X n + 1 via the probabilities assigned to the intervals ( t ( i ) , t ( i + 1 ) ) , ( r c ( j ) , t r c ( j ) ) and ( t l c ( w ) , l c ( w ) ) , where i = 1 , 2 , … , u , j = 1 , 2 , … , v and w = 1 , 2 , … , k , with t ( 0 ) = 0 and t ( u + 1 ) = ∞ . These probabilities can lead to derive lower and upper probabilities for any event of interest in terms of X n + 1 , by applying the same technique used for A ( n ) -based inference [8]. If we are interested in the event X n + 1 ∈ A , with A a set of the nonnegative real values, then the lower probability for this event, referred to by P ̲ ( X n + 1 ∈ A ) , is derived by summing only the probabilities for X n + 1 on intervals ( t ( i ) , t ( i + 1 ) ) , ( r c ( j ) , t r c ( j ) ) and ( t l c ( w ) , l c ( w ) ) , which are completely within the set A . The upper probability for the event X n + 1 ∈ A , referred to by P ¯ ( X n + 1 ∈ A ) , is derived by summing all the probabilities for X n + 1 on intervals ( t ( i ) , t ( i + 1 ) ) , ( r c ( j ) , t r c ( j ) ) and ( t l c ( w ) , l c ( w ) ) , which have a nonempty intersection with the set A .

6 Examples

In this section, we apply the generalizations of A ( n ) assumption to two data sets, including double-censored observations. In each example, two bounds are derived to predict the survival function for next future observation based on each suggested generalization.

6.1 Simulated example

Before applying the generalizations, it is worth to describe how to create a double-censored data set. The simulated data are formed by generating seven observations from Uniform distribution, with parameters a = 0 and b = 60 , and this is repeated three times. The first generated data are considered for event data, and the second and third generated samples are considered for right- and left-censored data. Then, we use equations (8) and (9) to define the double-censored data set. The data are presented in Table 1, which includes two right-censored observations with sign “+,” one left-censored observation with sign “ − , ” and four noncensored observations with no sign.

(8) x o = max [ min ( t o , r c o ) , l c o ] , for o = 1 , 2 , … , 7

(9) d o = 1 if x o = t o (uncensored) 2 if x o = r c o (right-censored) 3 if x o = l c o (left-censored) ,

where x o is the time and d o is the censored indicator.

Table 1

The double-censored data for simulated example

x o	13	17	22	29	35	46	52
d o	+	+					‒

Each generalization of A ( n ) assumption leads to a partially specified probability distribution, via the probabilities calculated by equations (6) and (7), for survival time of a future observation X n + 1 , which we refer to by a random quantity X 8 in the simulated example. The probabilities related to the intervals ( t ( i ) , t ( i + 1 ) ) , ( r c ( j ) , t r c ( j ) ) , and ( t l c ( w ) , l c ( w ) ) are presented in Tables 2 and 3, and these assignment probabilities can be used to derive bounds for the survival function of X 8 , so bounds for S X 8 ( t ) = P ( X 8 > t ) , for t ≥ 0 . We call the maximum lower bound “the lower survival function,” denoted by S ̲ X 8 ( t ) = P ̲ ( X 8 > t ) , and the minimum upper bound “the upper survival function,” denoted by S ¯ X 8 ( t ) = P ¯ ( X 8 > t ) . These imprecise probabilities are derived as described in Section 5.

Table 2

The generalization A of A ( n ) -based probabilities for X 8 on the intervals ( t ( i ) , t ( i + 1 ) )

Interval	Probability
(0, 22)	0.200
(22, 29)	0.200
(29, 35)	0.200
(35, 46)	0.200
(46, ∞ )	0.200

Table 3

The generalization B of A ( n ) -based probabilities for X 8 on the intervals ( t ( i ) , t ( i + 1 ) ) , ( r c ( j ) , t r c ( j ) ) , and ( t l c ( w ) , l c ( w ) )

Interval	Probability
(0, 22)	0.150
(13, 22)	0.025
(17, 22)	0.025
(22, 29)	0.200
(29, 35)	0.200
(35, 46)	0.200
(46, 52)	0.025
(46, ∞ )	0.175

The upper survival function for X n + 1 can be easily derived due to the fact that the probabilities defined based on the generalizations of A ( n ) assumption are all specified on the intervals ( t ( i ) , t ( i + 1 ) ) , ( r c ( j ) , t r c ( j ) ) , and ( t l c ( w ) , l c ( w ) ) . To derive S ¯ X 8 ( t ) = P ¯ ( X 8 > t ) , for t > 0 , we sum up all mass probabilities related to the intervals ( t ( i ) , t ( i + 1 ) ) , ( r c ( j ) , t r c ( j ) ) , and ( t l c ( w ) , l c ( w ) ) , which have intersections with the set ( t , ∞ ) . To compute the lower survival function for X n + 1 , S ̲ X 8 ( t ) = P ̲ ( X 8 > t ) , for t > 0 , we sum up all mass probabilities related to the intervals ( t ( i ) , t ( i + 1 ) ) , ( r c ( j ) , t r c ( j ) ) and ( t l c ( w ) , l c ( w ) ) , which are completely within the set ( t , ∞ ) . Both bounds of the survival function are step functions, and they both equal to one at time 0 due to the assumption that there are no observed events at time 0, t ( 0 ) = 0 . In the last interval, ( . , ∞ ) , the upper survival function bound is a positive constant, but the lower survival function bound is equal to zero.

Tables 4 and 5 present the survival function values for X 8 , specified on intervals created by the simulated data based on both generalizations of A ( n ) assumption, and the results are presented in Figure 1. The values of the lower and upper survival functions at observation times can be easily derived from Tables 4 and 5, with respect to the fact that the lower survival function is continuous from the left at all observations, and the upper survival function is continuous from the right at event times. The right-censored and left-censored observations increase the difference between corresponding upper and lower survival functions at any censored observation. In case of data including many censored observations, regardless whether they are right censored or left censored, the difference between corresponding bounds of the survival function at time t becomes large.

Table 4

The lower and upper survival functions for X 8 based on the generalization type A of A ( n ) assumption

t ∈ ( ⋅ , ⋅ )	S ̲ X 8 ( t )	S ¯ X 8 ( t )
(0, 22)	0.8	1
(22, 29)	0.6	0.8
(29, 35)	0.4	0.6
(35, 46)	0.2	0.4
(46, ∞ )	0	0.2

Table 5

The lower and upper survival functions for X 8 based on the generalization type B of A ( n ) assumption

t ∈ ( ⋅ , ⋅ )	S ̲ X 8 ( t )	S ¯ X 8 ( t )
(0, 13)	0.850	1
(13, 17)	0.825	1
(17, 22)	0.800	0.975
(22, 29)	0.600	0.800
(29, 35)	0.400	0.600
(35, 46)	0.200	0.400
(46, 52)	0.175	0.200
(52, ∞ )	0	0.175

Figure 1

The lower and upper survival functions for X 8 based on the generalizations of A ( n ) assumption.

In Figure 1, we compared the generalization type A of A ( n ) assumption to the generalization type B of A ( n ) assumption. Due to the fact that the generalization type A partitions the sample support into u + 1 intervals and all intervals are based on the event observations only, we have less steps in the corresponding lower and upper survival functions, which are indicated by red and blue step lines respectively, in comparison to the lower and upper survival functions found based on the generalization type B, which are indicated by orange and green step lines, respectively. From Figure 1, it is obvious that the lower survival function based on the generalization type A is less than (or equal) the lower survival function based on the generalization type B, and the upper survival function based on the generalization type A is larger than (or equal) the upper survival function based on the generalization type B. After the largest observation, the lower survival functions for X 8 are both equal to zero. However, the upper survival functions always get a positive constant, unless we restrict the data range for X 8 by choosing a finite upper limit. Before the first event time t 1 = 22 , the upper survival functions are both equal to one, but the lower survival functions are less than one, and they decrease at the censored observations, which is not the case with the upper survival functions. We believe that the generalizations of A ( n ) -based lower and upper survival functions for X 8 are better suited for such graphical presentations, as they indeed give a wider picture of the double-censored data, and from a predictive perspective, they can easily be interpreted. For the case of large data and many observations are event, not censored, these proposed methods become nearly identical, and this is obvious in the following example.

6.2 AIDS example

The data set is from a cohort of drug users recruited in a detoxification program in Badalona (Spain) [20]. On the basis of the data, we may predict the lower and upper survival functions for the elapsed time from starting IV-drugs to AIDS diagnosis for the next observation. The data of 232 patients infected with HIV are presented as follows; 136 left AIDS-free (right censored), 14 died with AIDS without prior diagnoses (left censored) and 82 had AIDS while in the program (noncensored).

Figure 2 presents the lower and upper survival functions based on the generalizations of A ( n ) assumption. It is rational that they are nearly identical and the differences between the lower and upper survival functions become small due to the large sample size with many not censored observations.

Figure 2

The lower and upper survival functions for X 1001 based on the generalizations of A ( n ) assumption.

7 Treating such cases of ties

For simplicity, we have assumed that there are no ties in the data sets through this article, but in real studies, ties could occur in different scenarios. There are seven kinds of ties that could occur: tied event observations, tied right-censoring observations, tied left-censoring observations, ties among event and right-censoring observations, ties among event and left-censoring observations, ties among left-censoring and right-censoring observations, and ties among event and left-censoring observations with right-censoring observations. With the first three situations, we break the tied observations by adding a small value to those ties. With the fourth situation, we assume that the right-censoring times occur after the event observations, where this assumption has been widely used in the literature [3,21]. With the fifth situation, we assume that the left-censoring times occur before the event observations. With the sixth situation, we assume that the right-censoring times occur after the left-censoring times. With the last situation, we assume that the right-censoring times occur after the event observations and the left-censoring times occur before the event times.

An interesting result of having tied event time observations, say at time t t , is that our method, like rc- A ( n ) -based inferences and A ( n ) -based inferences in general, then gives a positive predictive probability for X n + 1 to occur at t t . This seems rational due to the fact that more than one event happened at time t t supports the case that future events can also occur at the time t t . Of course, such ties have been widely occurred due to data display and the discrete nature of measurement [22].

8 Concluding remarks

The nonparametric predictive inferences with double-censored data for one future observation X n + 1 introduced in this article can be used in multiple different ways. Imprecise probabilities and survival functions, as presented in Sections 5 and 6, can be used for a variety of inferential predictive problems, where the predictive inferences have been presented in many statistical applications. Some references suggested the use of A ( n ) -based nonparametric predictive inference for real-valued data [9–12], and other references support the use of rc- A ( n ) -based nonparametric predictive inference for right-censored data [4,23–25]. For bivariate data, Coolen-Maturi et al. [26], Muhammad [27], and Muhammad et al. [28] introduced the use of bivariate-data- A ( n ) -based nonparametric predictive inference.

In this article, we emphasize to keep the generalizations of A ( n ) assumption with few mathematical assumptions and the inferences based on the proposed assumptions are suited for the situations, which we have vague knowledge about. Of course, the generalizations of A ( n ) -based inferences may not be able to lead to the optimal decisions, but they can be used as a basis for studying the effect of additional modeling assumptions on final inferences, or related decisions, in the case of wishing to use methods with more structure. For example, when comparing two survival groups predictively, in the case of data including many double-censored observations, the range of survival functions between the lower and upper bounds per group may lead to preference of either group. In such cases, the generalizations of A ( n ) -based inferences conclude that strong inferences may not be possible based only on the data, so there is a need to add further modeling assumptions or to add more observations. From this situation, the generalizations of A ( n ) -based inferences are related to robust statistical methods.

Another generalization of A ( n ) assumption can be proposed for data including double-censored observations by using the exponential distribution for the right-censored observations. This distribution has been widely used to model lifetime data including right-censored observation. The study by Wu [29] can be a good source to start this future work.

BinHimd [30], BinHimd and Coolen [31], and Coolen and BinHimd [32] show that the A ( n ) assumption can be used to provide a smoothed bootstrap method for real-valued data. For right-censored data and bivariate data, Al Luhayb [33], and Al Luhayb et al. [34,35] used the rc- A ( n ) and bivariate-data- A ( n ) assumptions to provide smoothed bootstrap methods; for more information, it is good to see [36,37]. The generalizations of A ( n ) assumption can smooth the bootstrap method for double-censored data, and the bootstrap method can be used for survival analysis and testing. These extensions will be left as future research topics.