Identification of in-sample positivity violations using regression trees: The PoRT algorithm

2.1 A conceptual overview

The PoRT algorithm looks for combinations of a small number of predictors, where the exposed and unexposed samples are highly imbalanced (almost all the observations belong to one of the two groups), and the overall size of the subgroup is sufficiently high (Figure 1). It involves a two-step procedure. First, PoRT learns a succession of RTs according to different combinations of predictors. Second, it identifies imbalanced subgroups using two reading hyperparameters.

Figure 1

Flowchart presenting the PoRT algorithm. P(A): Exposed and unexposed proportions; C: Binomial coefficient representing the maximal number of RTs for the step; S: Proportion of the whole sample contained in the subgroup; α, β, and γ were user-supplied hyperparameters.

2.2 How are the RTs learned?

The algorithm learned the RTs using the R package rpart to predict the exposure on different combinations of predictors [20]. We define the first hyperparameter γ, which refers to the maximum number of predictors used to define the nodes. We stress that the predictors set must match the variables used for the adjustment since positivity is only relevant for them [2]. In contrast, including pure predictors of the exposure in the adjustment set increases the risk of positivity violations. For instance, if γ = 1, each RT predicted the exposure on one different predictor. If γ = 2, each RT predicted the exposure on one different predictor and then on a different combination of two predictors, and so on (Figure 1). Three other hyperparameters, specific to rpart, must be defined. First, the minimal number of individuals in a node to make it split. Second, the minimal number of individuals in the leaves for the parent node to be split. Third, the maximum depth of the tree consists of the maximum number of successive splits. As in rpart, we refer to these hyperparameters as minsplit, minbucket, and maxdepth. Figure S2 illustrates the impact of these hyperparameters on the RT’s learning. The rules of splitting are nicely summarized in the study by Kang et al. [8].

2.3 How are the RTs read?

Consider (α, β) as the two user-defined parameters to define the positivity violation. Parameter α is the minimal proportion of the whole sample size to consider a problematic subgroup. Parameter β is the exposed or unexposed proportion under which one can consider an in-sample positivity violation [3]. A node will be considered problematic if the proportion of being (un)exposed is more extreme than β or 1 − β and if it contains at least α individuals of the whole sample (see Figure 2 for an illustration). Note that, once a node is flagged, its descendants are not considered to avoid uninformative nested subgroups (as shown in Figure 2b).

Figure 2

Impact of the hyperparameters α and β on the problematic subgroups’ identification for a hypothetic tree. (a) α = 0.05 and β = 0.05, (b) α = 0.05 and β = 0.10, (c) α = 0.01 and β = 0.05. The gray nodes correspond to subgroups violating the positivity assumption according to the parameters α and β. P(A): Exposed or unexposed proportions, S: Proportion of the whole sample contained in the subgroup.

2.4 The algorithm definition

On the basis of hyperparameters previously defined, we used RTs to identify problematic subgroups. Figure 1 presents an overview of the proposed algorithm. The first step consists of learning one tree for each predictor and memorizing the nodes corresponding to problematic subgroups (which can be leaves or intermediate nodes). If γ = 1, the algorithm stops. Otherwise, if at least one problematic subgroup is identified in the first step, the corresponding predictor(s) is(are) not considered in the next step, which estimates one tree for all possible couples of remaining predictors and memorizes the nodes corresponding to problematic subgroups. If γ = 2, the algorithm stops; otherwise, the third step consists of building one tree for all possible trios of remaining covariates not involved in the previously identified subgroups, and so on, until one reaches the value of γ.

2.5 The default values of the hyperparameters

To the best of our knowledge, there is no consensus on the precise definition of a subgroup presenting a positivity violation. Nevertheless, one can set α = 5% as several authors suggest trimming the weights at this level in inverse probability weighting analyses [21,22,23]. Following D’Amour et al., one can also set β = 5% [3]. Note that this value is consistent with the propensity score literature [4,24]. By default, the last parameter is set to γ = 2 as positivity violations defined by three or more variables may not represent real practices. However, this could change depending on the field of study. Note that, to build the RTs, we used the default values defined in the R package rpart (version 4.1–15): at least 20 individuals for intermediate nodes, at least six individuals for leaves, and a maximum depth of 30 [20]. The trees were then not pruned to obtain the largest number of divisions.

The α hyperparameter could be increased as the sample size decreases. Indeed, random violations are more likely to occur in such situations. Similarly, alternative values can be considered for the β hyperparameter. For instance, Petersen et al. [4] and Crump et al. [25] argued in favor of β = 1 and 10%, respectively. Note that exposure prevalence can also influence this choice, with a small prevalence leading to a tighter definition of positivity and thus a smaller value of β. The parameter γ may also be increased with the sample size to refine the definition of subgroups. The rpart’s hyperparameters for splitting were set to small values to obtain the largest number of possible divisions. This limits the risk of missing some problematic subgroups due to a too restrictive splitting, but increases the risk of false positive results.

3.1 Barbiturates during intensive care for patients with traumatic brain injury

Léger et al. investigated the impact of barbiturates on mortality using a cohort composed of 1,088 patients admitted to intensive care units for traumatic brain injury [26]. This treatment may be offered to reduce patients’ intracranial hypertension (IH) and its consequences in terms of brain damage. The authors excluded individuals older than 70 years, for whom therapy was contraindicated. We identified nine subgroups potentially associated with in-sample positivity violations (Table 1). First, the PoRT algorithm confirmed patients older than 75 years as being problematic, a higher threshold than that suggested by the authors. Second, it indicated that patients without osmotherapy at admission had a relative frequency of barbiturates lower than 5%. This seems clinically relevant since barbiturates are a last-line therapy and should be proposed only after osmotherapy. Third, PoRT identified four subgroups composed of patients without IH at admission. Indeed, barbiturates are rarely proposed from a preventive perspective for high-risk patients. Note that this characteristic was not identified in the first iteration of PoRT because the barbiturates’ relative frequency for patients without IH was 5.1%, only slightly above the 5% threshold. Among the three major violations with respect to the sample size, we found a substantial overlap between the patients (n = 534). They represent a broader subpopulation that should not be targeted in the study. Fourth, the algorithm detected that a lactatemia lower than 1 mmol/L, a simplified acute physiology score (SAPS) II score from 40 to 44 without severe trauma, and a SAPS II score from 25 to 55 along with a creatinine level between 50 and 60 mmol/L were associated with a relative frequency of barbiturates below 5%.

Figure 3 describes the results of the initial analysis performed by the authors and our reanalysis based on the restricted sample without the three previous subgroups associated with structural violations. In the initial study, the odds ratio (OR) was 2.2 (95% CI from 1.1 to 4.6). The exclusion of individuals without IH at admission reduced the sample size by almost 70%. The main impact was on the control group: the sample size decreased from 964 to 100 patients in this group versus 124 to 73 in the group under barbiturates. In the restricted sample (N = 173), the OR was 1.9 (95% CI from 1.0 to 3.5). The PSs for the discarded individuals ranged from 0.006 to 0.327, and more than 25% of these individuals presented a PS greater than the threshold of 0.05. In contrast, no individuals with a PS higher than 0.95 or lower than 0.05 (55.6% of the whole dataset) remained in the restricted dataset (i.e., after removing the problematic subgroups).

Figure 3

Comparisons of the results reported in the initial studies and those obtained by reanalyzing the data by excluding the patients associated with a structural violation of positivity. Léger et al. presented an OR, while the others presented a HR.

3.2 Kidney transplantations from marginal donors

Querard et al. compared grafts from standard and marginal donors, as defined by the expanded donor criteria [27]. Because of the shortage of kidneys for transplantation, kidney grafts harvested from standard donors are preferentially attributed to young recipients, resulting in a susceptible positivity issue. The authors did not consider recipient age among the eligibility criteria. In contrast, PoRT detected a problematic subgroup consisting of 391 recipients younger than 30 years (8.1% of the whole sample) with a marginal graft’s relative frequency of 3.1% (Table S2).

The exclusion of these individuals did not change the effect size (Figure 3). In the restricted sample (N = 4,442), the hazard ratio (HR) was 1.3 (95% CI from 1.2 to 1.5). In the initial study (N = 4,833), the HR was 1.3 (95% CI from 1.1 to 1.6). The PSs of the discarded individuals ranged from 0.002 to 0.024, while 16.8% had an extreme PS, and 8.8% remained after discarding the problematic subgroups.

3.3 Hypothermic perfusion machine for marginal donors

Foucher et al. compared the use of a hypothermic perfusion machine to static cold storage in kidney transplantations from expanded donors [28]. The authors reduced the studied cohort to individuals older than 45 years because of a potential structural violation: the old-to-old graft allocation policy results in a lower susceptibility of younger candidates receiving marginal grafts. The PoRT algorithm did not detect this issue when using the entire cohort with no restriction on patient age (N = 1,978). Indeed, 32 (3.3%) and 44 (4.0%) individuals were younger than 45 years in the perfusion machine and cold storage groups, respectively. In contrast, PoRT suggested two problematic subgroups. First, individuals receiving transplants before 2014 in four centers had a prevalence of perfusion machines ranging from 0.8 to 4.7%. This violation appeared plausible because some hospitals were slow to adopt hypothermic perfusion machines, which were only introduced in 2010 in France. Second, patients with a cold ischemia time under 20 h who received a transplant before 2013 (N = 101) received a graft under static cold storage with a relative frequency lower than 5%, which also appeared plausible because the machines were preferentially attributed to situations with a long transport from donors to recipients. This preferential attribution became less strict with the increase in the availability of machines over time.

The exclusion of these two problematic subgroups with the inclusion of all recipients regardless of age shifted the HR from 0.9 (95% CI from 0.6 to 1.3) in the initial study to 0.8 (95% CI from 0.6 to 1.1) in the novel targeted population. The PSs of the discarded individuals ranged from 0.606 to 0.993, and more than 50% of these individuals presented a PS lower than the threshold of 0.95. In contrast, 11.9% of the individuals have an extreme PS and 4.6% remain after discarding the problematic subgroups.

3.4 Induction therapy in elderly kidney transplant recipients

Masset et al. compared the risk of several adverse events following a kidney transplant in elderly recipients depending on their induction therapy: antithymocyte globulins versus basiliximab [29]. We did not detect any positivity violations with PoRT, regardless of the event and adjustment set, confirming the authors’ statements.

3.5 Sensitivity analyses: Impact of the α and β hyperparameters variation

As expected, lowering the value of parameter β decreased the number of subgroups identified (β = 1%, Table S3). In the study by Léger et al. [26], the problematic subgroups involving patient age and osmotherapy were no longer identified. Nevertheless, combinations of age, osmotherapy, and others were present, similar to the subgroups involving the IH in the main analysis, alerting against broader violations undetected with the current parameters. The subgroups identified in the studies by Querard et al. [27] and Foucher et al. [28] were also no longer identified due to the stringent definition of positivity.

With a broader definition (β = 10%, Table S4), we identified 11 new subgroups across the four studies. While their majority did not seem to have a plausible clinical interpretation, we identified the aforementioned subgroup of individuals without IH in the study by Léger et al. [26]. Importantly, such a choice can lead to the identification of broader subgroups for continuous variables, possibly incorrectly. For instance, we identified two subgroups in the study by Querard et al. [27] for recipients younger than 50 years and older than or equal to 65 years. These thresholds appear less plausible than that of 30 years.

We identified 6, 14, and 4 subgroups, leading to in-sample violations with parameter α at 1% in the studies by Léger et al. [26], Foucher et al. [28], and Masset et al. [29] (Table S5). Interestingly, in the study by Foucher et al. [28], recipient’s age lower than 50 years was identified as problematic when associated with transplantation before 2013. This result might explain the exclusion criteria used by these authors.

Unexpectedly, a higher α did not always reduce the number of subgroups identified (Table S6). Four new subgroups were identified in the study by Léger et al. [26], notably for combinations including age. This finding can be explained by the fact that PoRT cannot identify the absence of IH with a β at 5%, but it could when β is set at 1%. The threshold for age at 5% was 75 years, which represents 6.7% of the whole sample. Because PoRT could not find this subgroup by searching those including at least 10% of the whole sample, it identified several other subgroups defined by age. This again alerts against an underlying broader violation undetected with the current parameters.

3.6 Comparison with an RT using all predictors at once

When using an RT with all the predictors at once, only one node was identified as problematic in the study by Léger et al. [26]. It corresponded to the absence of osmotherapy at inclusion (Table S7). For the study by Foucher et al. [28], this approach led to one problematic subgroup, also identified by our approach. The second subgroup identified by our approach remained unidentified. In the studies by Querard et al. [27] and Masset et al. [29], no subgroup was identified as problematic: the two approaches resulted in the same conclusions. Trees with the entire set of predictors were presented in Figures S3–S5.

The tree for the first study (Figure S3) presented several additional (potentially finer-grained) subgroups, like the individuals without osmotherapy nor IH at inclusion (or all descendants of this subgroup but one). However, the other main subgroup without IH at inclusion (with osmotherapy) is not identified. Since the target population was the individuals with IH, individuals without IH must be identified regardless of their osmotherapy status. By building only one tree with the entire adjustment set, one becomes highly dependent on the first splits, which may not be the most informative for the positivity. Moreover, considering the descendants of a previously flagged node increases the number of false positives and complicates the interpretation of the results.

3.7 Impact of a prior continuous categorization of predictors

Without prior categorization, PoRT detected seven problematic subgroups in the study by Léger et al. [26], including three considered clinically plausible (Table S8). In contrast, categorizing continuous predictors using clinically meaningful thresholds led to the identification of nine problematic subgroups, including six plausible according to domain experts (Table 1). In the study by Querard et al. [27], the subgroup identified in the main experiment was flagged again but with a threshold higher of 4 years. In the study of Foucher et al. [28], only one subgroup was identified out of the two in the main experiment. The missing subgroup involved the cold ischemia time, a variable challenging to split algorithmically due to its distribution. In the study by Masset et al. [29], PoRT identified three new problematic subgroups across the seven different datasets. However, these subgroups involve variables categorized automatically at uninterpretable thresholds. Therefore, categorizing continuous predictors reduced the risk of false positives by identifying subgroups more coherent with clinical practice.