An Instrumental Variables Design for the Effect of Emergency General Surgery

3.1 The instrument: Physician preference for operative management

We use a physician’s preference for operative management as an instrument for EGS. We rely on the observation that surgeons demonstrate differing preferences for operative versus non-operative care. Our instrument is an example of a broad class of instruments based on a physician’s treatment preferences, which have been widely used to study the effectiveness of drugs (Brookhart and Schneeweiss 2007). To our knowledge, this is the first use of a preference-based instrument defined at the level of a surgeon. Like all preference-based instruments, our measured instrument does not have a direct causal effect on D, since it is a proxy for the physician’s true preference for operative care. Thus our measured IV is a surrogate IV (Hernán and Robins 2006).

We used random sample splitting to separate the measurement of the instrument from the use of the instrument in the data analysis. This avoids bias from using the same data twice (Bound et al. 1995). For each surgeon, we randomly split his or her patient population in half. Using one half of the data, we calculated the proportion of times a surgeon operated for the population of patients they treated with the 51 specific acute conditions. This measure serves as the instrument for the remaining patient population. For the surgeons in our data, this measure expressed as percentage varied from less than 1% to 100%, with an average of 59%. As such, there is wide variability in preferences for operative care across surgeons. Often instruments of this type are defined at the hospital level or geographically (Brooks et al. 2003; Stukel et al. 2007). We could easily measure the instrument at the hospital level. The data, however, provide evidence that this is not an optimal measurement strategy. If we measure the instrument at the hospital level, the measure varied from 1% to 38% with an average of 16%. As such, the variation in the instrument is much compressed. This is not surprising, since the decision to operate in a given emergent admission for emergent admissions primarily resides with the individual surgeon. Moreover, the correlation between the instrument and the outcome is also weaker at the hospital level. Thus defining the instrument at the hospital level obscures considerable variation in whether a patient is treated operatively or not.

3.2 Causal assumptions

Next, we review and assess the necessary assumptions for our IV to identify the effect of D on Y. We begin with general causal assumptions and then proceed to the specific IV assumptions.

First, we assume that the stable unit treatment value assumption (SUTVA) holds (Rubin 1986). SUTVA is comprised of the two following components: (1) the treatment levels of D (1 and 0) adequately represent all versions of the treatment and 2) a subject’s outcomes are not affected by other subjects’ exposures. The first component of SUTVA is often referred to as the consistency assumption in the epidemiology literature. In our study, D represents a wide variety of surgical procedures, from the removal of an appendix to wound debridement. However, as we outline below, we restrict comparisons within surgical conditions such that within each subgroup of patients the possible treatment options are limited and highly similar. That is, while there may be some variation in how patients receive an appendectomy, we assume that this variation in treatment corresponds to the same potential outcomes. The second component of SUTVA rules out a subject’s outcomes being affected by other subjects’ exposures. In general, it is difficult to conceive of how the selection of surgical versus non-surgical care for one patient could affect outcomes for other patients even within the same hospital. One possibility is through resource constraints. Surgeons and emergency operating rooms are a limited resource, and whether one patient receives surgery could be affected by factors related to resource availability or access. However, in most cases resource constraints would only result in a delay of surgery instead of switching a patient to non-operative management.

Next, we turn to specific IV assumptions. Identification in the IV framework typically relies on the following four assumptions: (1) ignorable (as-if random) assignment of the instrument to patients; (2) the instrument must have a nonzero effect on the exposure; (3) no direct effect of the instrument on the outcome, also known as the exclusion restriction; and (4) monotonicity (Angrist et al. 1996).

Ignorability.    First, it must be the case that patients are assigned surgeons with high or low preferences for operative care in an as-if random fashion. That is, care from a surgeon with a higher preference for surgery should tell us nothing of clinical relevance about that surgeon’s patients. The fact that patients are receiving emergency care bolsters this assumption, since it is unlikely that patients are paired with a surgeon based on that doctor’s preference for operative treatments in an acute care setting. We assessed whether assignment to the instrument appears as-if random by calculating covariate balance for patients cared for by surgeons above and below the median preference for operative care. Note that this is only a partial assessment of this assumption, since we cannot test whether unobservables are balanced. Moreover, while the IV assumption does not strictly imply balance on observables, such diagnostics are widely recommended (Baiocchi et al. 2012; 2013).

Table 1 contains means and standardized differences for selected baseline covariates. The standardized difference is the difference in means divided by the pooled standard deviation. A standardized difference of one implies that the difference in means is equal to one standard deviation. Table 1 also contains balance statistics for whether patients received EGS or not. In almost every case, we find that standardized differences are lower when stratifying by the IV rather than the exposure, which implies that allocation of physician preference appears more as-if random than the allocation of EGS. Of the variables recorded in Table 1 we might be most concerned about measures that are proxies for surgical skill. That is, we might be particularly concerned that those surgeons that prefer to operate are more skilled. The results in Table 1 do little to suggest that is the case. That is, surgeons with a higher preference for operating are very similar to surgeons with a lower preference for operating in terms of age and experience. Even more striking is that surgeons with a higher preference of operating had nearly identical levels of mortality and prolonged lengths of stay for past procedures. Thus, at best, our measures of surgical skill are weakly correlated with our proposed instrument. The appendix contains full balance statistics for all measured covariates.

Next, we use bias ratios to further assess whether an IV design may have less bias than a design based on RA (Brookhart and Schneeweiss 2007; Baiocchi et al. 2014). For this analysis, we first calculate prevalence differences under each design. The prevalence difference under the IV design is defined as the difference in means for baseline covariates across a binary measure of the IV. We created two binary measures of the IV. The first stratifies the patient population at the median value of the IV. The second compares patient covariates for surgeons in the top quartile of the IV to patient covariates for surgeons in the 25th to 75th percentile of the IV. The prevalence difference under an RA design is the difference in means for baseline covariates across levels of the exposure – undergoing EGS or not. Following Baiocchi et al. (2014), we then calculated bias ratios as the IV prevalence difference divided by strength of the IV divided by the RA prevalence difference. Bias ratios of less than one indicate less bias associated with the IV design as compared to RA (Baiocchi et al. 2014). For 110 measured confounders, the IV design had less bias for 77% of the covariates using the median split for the IV. Using the split at the upper quartile, the IV design had less bias for 62% of the covariates.

We performed additional analyses to explore the relative differences in balance between the IV and RA approaches. First, we plotted scaled and unscaled covariate balance estimates with confidence intervals (Davies 2015; Jackson and Swanson 2015). Figure 1 contains these plots for key covariates. The plot clearly demonstrates that the differences are relatively small and the confidence intervals typically overlap. Plots for the full set of covariates are included in the appendix. We also calculated tested whether these differences were significantly different following Davies et al. (2017). We found that in many cases the differences were not statistically significant. See the appendix for results of this analysis. However, these diagnostic plots also show that measures of surgical skill are at best weakly associated with the proposed instrument.

Figure 1:

Bias plots for selected covariates.

IV Strength.    Next, we evaluated whether the instrument is correlated with the exposure. When patients have a physician above the median in terms of his or her preference for surgery, they are managed operatively 76% of the time. When patients have a physician below the median in his or her preference for surgery, they receive operative care 38% of the time. We also conducted a weak instrument test; the F-statistic is 29,000, well above the critical value threshold of 16.4 (Stock and Yogo 2005).

Exclusion Restriction.    Next, it must the the case that any effect of a surgeon’s propensity to operate on the outcome is only a consequence of the medical effects of operative management. A violation of this assumption would occur if receipt of care from a surgeon with a strong preference for surgery includes other aspects of care that might affect outcomes. For example, if nursing care provided to patients of high preference surgeons was superior to that provided to patients of low preference surgeons, this would violate the exclusion restriction. We bolster this assumption by comparing patients who receive care in the same hospital. This should hold fixed other system factors of care that might affect outcomes. For patients within the same hospital, it is unlikely that a surgeon’s preference for operative care will have any affect on outcomes except through receipt of EGS. See below for details on how we used matching to implement within hospital comparisons in the statistical analysis.

Monotonicity.    In an IV design, subjects fall into four different classes: compliers, always-takers, never-takers, and defiers (Angrist et al. 1996). Compliers are patients who are exposed to surgery if they were encouraged by the IV but would not otherwise have surgery. Always-takers have surgery regardless of assigned IV status, and never-takers are never exposed to surgery regardless of IV status. Defiers are patients who are only exposed to the treatment (surgery) if not encouraged or are unexposed if encouraged. If no defiers are present, the IV estimate is well-defined for compliers. Many IV designs in epidemiology assume that no patients are defiers, which is generally referred to as the monotonicity assumption.

Recent work has critiqued the monotonicity assumption in the context of preference-based instruments (Swanson and Hernán, 2014, 2017). Monotonicity violations are likely when the instrument is not delivered in a uniform way to all subjects. This occurs with preference-based instruments since the delivery of the preference to patients varies across physicians. In our context, for the monotonicity assumption to hold it must be the case that there are no patients who would receive surgery when seen by a physician who usually does not prefer surgery but would not be given surgery when seen by a physician who usually prefers surgery.

In general, since a surgeons’ preferences represent weighing a variety of risks and benefits, there may be opportunities for a patient to be treated contrary to a physicians’ preferences. As an example, assume we have two surgeons who work in the same hospital. Surgeon A generally prefers to operate but makes exceptions for hernia patients (because of some known contraindications). Surgeon B generally tries to avoid surgical treatments but makes exceptions for patients who are very healthy and can withstand the invasive nature of many procedures. Thus any patient that has a hernia but is also very healthy would be treated in a way contrary to each surgeons’ general preferences for operative care and is a defier.

Given the implausibility of the monotonicity assumption in our design, we adopt the framework of stochastic monotonicity (Small et al. 2017). The stochastic monotonicity assumption has two components. First, Z must be independent of the potential outcomes for Y and unobserved common causes of Z and Y. Second, if we stratify on all measured and unmeasured confounders of D and Y, then within each stratum there are more compliers than defiers. How does this translate into our context?

We must assume that for patients with similar unmeasured common causes of D and Y, the probability of surgery is at least as high when treated by a surgeon with a strong preference for operative care as it is for patients seen by a surgeon with a weak preference for operative care. This assumptions appears reasonable. First, the primary unmeasured common causes of D and Y are the health status of the patient before surgery. Thus for stochastic monotonicity to hold, it must be true that for patients with similar unobserved risk indicators, the probability of surgery is the same or higher when they are treated by a surgeon who prefers to operate as it is when they are treated by a surgeon that does not prefer to operate. Given that many surgeons have a strong preference for operative care, this would seem highly plausible.

Invoking the stochastic monotonicity assumption does not change our estimation strategy. Small et al. (2017) show that the conditional and unconditional Wald estimator remains a consistent estimator under stochastic monotonicity. However, the target causal estimand differs. If we had invoked deterministic monotonicity, the target causal estimand would be the effect of surgery among the compliers known as the local average treatment effect (LATE) in the subpopulation of compliers (Angrist et al. 1996). Under stochastic monotonicity, the IV estimand is a weighted average among the subgroups for whom the IV has a stronger effect get more weight. For example, as we noted above, we defined 9 surgical conditions that can be managed with operative and non-operative treatments. If surgeon preference for operative care has a stronger effect among those patients with one of the nine conditions, this subgroup will receive a greater weight in the IV estimate. However, these subgroups are not known a priori. We can seek to identify them by finding groups where the IV has the strongest effect. Below we conduct a sensitivity analysis to understand how the IV estimates might generalize to the larger patient population.

4.1 Matching

We used matching to both perform risk adjustment and increase the plausibility of the IV design. As such, we implement two separate matches. One we designate as the RA match, and one we designate as the IV match. In the RA match, we match by EGS status, and in the IV match, we match by IV status. For the IV design, we applied near-far matching. Near-far matching pairs patients such that differences in observed confounders are minimized while the contrast in instrument values is maximized (Baiocchi et al. 2010). In our application, this will produce matched pairs of patients that are similar in terms of baseline covariates, but less similar in terms of the physician’s preference for operative care. A near-far match will increase comparability while further strengthening the instrument. IV designs based on stronger instruments are more resistant to bias, and near-far matching allows us to further increase the strength of our instrument (Small and Rosenbaum 2008).

For both the RA and IV match, we applied the following matching constraints. First, we matched patients exactly within hospital. Exactly matching within hospitals controls for any relevant hospital level factors that contribute to patient level outcomes, and implies that the level and overall quality of care outside of surgery should be similar, which increases the plausibility of the exclusion restriction. Furthermore, matching within the hospitals removes the effects of potential differences in claims coding across hospitals. We also exactly matched on the nine indicators for surgical condition.

We applied near-fine balance constraints to the 51 indicators for surgical sub-conditions. Fine balance constrains two groups to be balanced on a particular variable without restricting matching on the variable within individual pairs (Rosenbaum et al. 2007). A match with fine balance on the surgical sub-conditions produces patients paired on the IV with identical marginal distributions for these indicators. Rosenbaum et al. (2007) uses the term “fine balanced” since a variable has been balanced in terms of the marginal distribution, but units are not exactly matched on the variable. A near-fine balance constraint returns a finely balanced match when one is feasible, and otherwise minimizes the deviation from fine balance (Yang et al. 2012). Thus by applying near-fine balance to the surgical sub-conditions, we place no restriction on individual matched pairs–any one treated subject can be matched to any one control, but the marginal distribution of these indicators will be exactly or nearly exactly the same across levels of the IV. We implemented fine balance via a matching algorithm that finely balances the joint set of interactions among these variables (Pimentel et al. 2015). For the remaining covariates, we minimized the total of the within-pair distances on covariates as measured by the Mahalanobis distance (Mahalanobis 1936; Rubin 1980). Finally, we also applied optimal subsetting to discard matched pairs with high levels of imbalance on covariates (Rosenbaum 2012).

To assess the quality of the matches, we used the standardized difference after matching, which is calculated for a given covariate as the mean difference between matched patients divided by the pooled standard deviation before matching (Cochran and Rubin 1973). In the match, we attempted to produce standardized differences of less than 0.10, the recommended threshold (Silber et al. 2001; Rosenbaum 2010). We found that balancing all 110 covariates to this standard caused a considerable loss of sample size. We increased the sample size using the following analytic strategy. First, we improved the balance until fewer than 20 covariates had standardized differences of less than 0.30. For these covariates we applied additional covariate adjustment via regression. We settled on a match with 16 covariates that had standard differences larger than 0.09. The largest of these was 0.23 and 14 of the covariates had standardized differences between 0.10 and 0.14. The appendix outlines an additional analytic strategy we used for the measures of surgical skill.

4.2 Outcome estimates

For the IV design, we estimated risk differences applying the Wald estimator via two-stage least squares after matching. For the RA design, we estimated risk differences using linear probability models. After matching, we also applied the weak instrument test (Stock and Yogo 2005). We corrected standard errors for clustering within surgeons under both RA and IV. We also sought to understand whether effects varied with condition type. We performed subgroup analyses for each of the nine major acute surgical condition types. To perform the subgroup analyses, after matching, outcomes were compared separately within each condition type to understand whether differences between operative and non-operative care differed by condition risk.

As a robustness check, we also estimated IV results without matching. For these estimates, we used two-stage least squares only, conditioning on the same set of measured covariates. For these models, we also included fixed effects for hospital, the nine general conditions, and the 51 specific sub-conditions. We report these estimates in the appendix as they generally agreed with estimates based on matching.

5.1 Sensitivity analyses

Finally, we perform several different sensitivity analyses to understand whether our conclusions would be altered if we violated key assumptions. Following the recommendation in Swanson and Hernán (2013), we report nonparametric bounds for the IV estimate (Balke and Pearl 1997). These bounds relax the assumption of no defiers and the exclusion restriction. The nonparametric bounds for mortality are – 39% and 22%, which are consistent with both a strong harmful and protective effect. The bounds demonstrate that a considerable amount of information is provided by the two key IV assumptions. However, nonparametric bounds for mortality based on the average treatment effect using the method of Manski (1990) are wider: – 61% and 39%.

Next, to understand whether the estimated effect of EGS based on IV is possibly the result of an unobserved confounder between the instrument and the outcome, we apply a form of sensitivity analysis outlined in Small and Rosenbaum (2008). The sensitivity analysis indicated that the mortality finding would remain statistically significant in the presence of a confounder that increased the odds of both EGS and higher mortality by 5%. Thus our estimates are relatively sensitive to a hidden confounder. For vascular procedures, an unmeasured confounder would have to increase the odds of both EGS and higher costs by 220%. For vascular conditions, it would take an substantial amount of unmeasured confounding to qualitatively change our conclusions.

Next, we perform a sensitivity analysis that allows us to understand whether the estimates would change as the proportion of defiers increases (Small et al. 2017). Under this method, we calculate bounds on the IV estimate as the possible percentage of the defiers increases. We found that these bounds for the IV estimate did not include zero until at least 8% of the study population are defiers. The lower bound on the 95% confidence interval for this estimate did not include zero unless 4% of the population are defiers. To understand whether these are large or small values, we estimated the fraction of defiers in our data using the methods in Small et al. (2017). We estimated this quantity by fitting logistic regressions of D on X for the Z = 1 subjects and D on X for the Z = 0 subjects. We fitted these models to the matched data, and specified X as the covariates identified for additional adjustment after matching. We used these estimates and the full data to predict the probability of D = 1 under high and low values of the instrument. We then calculated the proportion of subjects, conditional on the covariates, that were exposed contrary to their instrument value. This estimated proportion of defiers in our data is 0. See the appendix for details on this sensitivity analysis.

Finally, we can also bound the global average treatment effect as a function of the extent of treatment effect heterogeneity – i.e. variation in the IV estimate due to patient population characteristics (Small et al. 2017). We calculated bounds for the global average treatment effect as a function of the difference between the average treatment effect for the always-takers and never-takers.

Table 4 contains the bounds on the global ATE as well as bootstrapped 95% confidence intervals. For a moderate level of effect heterogeneity the global treatment effect could be as small as 0.48% or as large as 4.9%. In general, the bounds are more closely aligned with a larger harmful effect in the full population.