Spillover detection for donor selection in synthetic control models

2.1 Identifiability of SCs

Historically, SC identifiability relied on assuming that the data generating process can be modelled as a latent linear factor model. With this assumption, the counterfactual is identified as a linear combination of valid donors. For instance, see literature [11–13], and extensions of these approaches utilizing Bayesian structural time-series by Brodersen et al. [14].

More recently, Shi et al. [21] argued that linearity emerges in a non-parametric manner if the target and donor units are in fact aggregations of “smaller” units (e.g. country-level data are aggregates of individual-level behaviours). The need for this “aggregate unit” assumption was subsequently removed by Shi et al. [20] and Zeitler et al. [18], who leveraged proximal causal inference to prove that the counterfactual can be non-parametrically identified as a (potentially non-linear) function of valid donors.

Zeitler et al. [18] introduced invariant causal mechanisms as a necessary condition for SC non-parametric identifiability. We define this condition in Section 3.1, and make use of it throughout the rest of the article. How does this relate to previous works on SC? The original SC formulation of Abadie et al. [12] assumes that the data generating process follows a latent linear factor model with time-independent factor loadings. Time-independence of the factor loadings is a special case of the more general invariant causal mechanism assumption employed here. Shi et al. [20] do not explicitly assume that causal mechanisms are invariant, however they do assume the existence of a time-independent confounding bridge function mapping donor outcomes to target outcomes. Zeitler et al. [18] showed that the time-independence of this confounding bridge function can in fact be understood as a consequence of invariant causal mechanisms.

2.2 Proximal causal inference

Proximal causal inference was initially investigated by Kuroki and Pearl [22] and Miao et al. [23], and has been further developed by Tchetgen et al. [24]. It has been used, for instance, in long-term causal effect estimation by Imbens et al. [25], and formulated in terms of the graphical causal inference framework by Shpitser et al. [26]. In the context of SC models, Shi et al. [20] and Zeitler et al. [18] employed proximal causal inference to prove non-parametric identifiability.

Recently, Liu et al. [27] proposed a novel SC model leveraging special donors called “surrogates,” which can include potentially invalid donors. These surrogates differ from the usual donors in that they are correlates of the causal effect instead of the target outcome. For example, in the simple case where the causal effect is a linear function of some latent variable w t such that τ t = γ w t + ε τ t , then an observed variable s t = ρ w t + ε s t is a valid surrogate. Assuming the existence of such surrogates (as well as the requisite domain knowledge to identify them), Liu et al. [27] demonstrated SC estimation based on post-intervention data alone.

2.3 Sensitivity analysis

Later in the article, we discuss scenarios where our donor selection procedure fails, and relate these to recent works on sensitivity analysis in SC [18] and negative control [19]. Sensitivity analysis in the causal inference literature has mainly been focused on investigating omitted variable bias in propensity-based models. This line of work originated in the studies of Rosenbaum and Rubin [28] and Imbens [29], with modern formulations provided by previous studies [30–32].

the studies of Rosenbaum and Rubin [28] and Imbens [29]

In the context of SC models, sensitivity analysis has been investigated by Zeitler et al. [18] and Nazaret et al. [33]. The work by Zeitler et al. [18] explored a formal framework for sensitivity to violations of identifiability of the full SC model, in particular, sensitivity to the existence of relevant latent variables with no observed proxies. Nazaret et al. [33], on the other hand, explored mis-specifications to the standard linearity assumption in SC models. Both of these works assumed valid donors (i.e. no donors impacted by spillover effects from the intervention).

Miao [19] investigated sensitivity analysis in the context of negative control (widely used in epidemiological research, negative control variables can be viewed as specific proxy types within the proximal causal inference framework [24,34]). In particular, Miao [19] introduced positive control outcomes (which are reminiscent of invalid donors in SC models), and discussed treating the (unknown) spillover effects as sensitivity parameters to evaluate the plausibility of causal estimates.

3.1 SC structural causal model (SCM)

In Definition 3.1  (first provided by Zeitler et al. [18]), we formally define SC models in the SCM framework of Pearl [35].

Figure 1

(a) SC DAG. Grey nodes are observed variables, white latent. The intervention I is applied at time point t , and is taken to be 0 for all time points before t , and 1 for all time points from t onwards. The noise terms for the target Y , donors X , and latents U have been suppressed for ease of exposition. Note that autocorrelation in the target time-series between y t − 1 and y t , or between x i t − 1 and x i t in the time-series for donor x i arises due to the causal links between the latents at different time points. As discussed in Appendix A, the model and results can be readily generalized to the case where the evolution depends on T time points. (b) Donor forecast. At time point t − 1 , the donors x 1 t − 1 , … , x N t − 1 are proxies for the latents u 1 t − 1 , … , u M t − 1 , which allows us to write x i t as a function of x 1 t − 1 , … , x N t − 1 and the noise terms at time t . The noise terms at time t − 1 have been suppressed for ease of exposition.

For simplicity, we sometimes follow [36] and suppress the functional dependence on the exogenous error terms for the latents u 1 , … , u M . We also often drop the indices on the donors and latents as follows: x t ≔ x 1 t , … , x N t , u t ≔ u 1 t , … , u M t , ε x t ≔ ε x 1 t , … , ε x N t , and ε u t ≔ ε u 1 t , … , ε u M t .

This definition of SCSCMs generalizes the standard latent linear factor model formulation of SCs [12]. In particular, notice that the target y t and donors x t can be arbitrary functions of the latents u t . We sometimes also use y t and x t to denote the values these variables take; however, the difference will be clear from the context. Note that autocorrelation in the target and donor time-series arises due to the causal links between the latents at different time points. As discussed in Appendix A, the definition of SCSCMs and our results can be readily generalized to the case where evolution depends on T time points.

We finish this section by defining the proxy variable completeness condition (Definition 3.2), and invariant causal mechanisms (Definition 3.3), which are necessary for SC non-parametric identifiability, and form the basis for our Theorem 3.1.

The following completeness condition formally defines when a set of donors can be considered proxies for a set of latent variables [18,24].

At a given time point, the completeness condition characterizes how much “information” the donors have about the latent variables, in the sense that any variation in the u ’s is captured by variation in x ’s. For the rest of the article, we assume that the set of donors x 1 t , … , x N t can be treated as proxies for the latents u 1 t , … , u M t .

A causal mechanism is the deterministic function that uniquely specifies a variable from its parents in the causal graph and is equivalent to the conditional distribution of that variable given its (latent and observed) parents. We assume that causal mechanisms are invariant (Definition 3.3).

In practice, this assumption means that we drop the time index from the causal mechanisms in the structural equations (Definition 3.1) such that u j t = m j ( u j t − 1 , ε u j t ) , x i t = f i ( u 1 t , … , u M t , ε x i t ) , and y t = g ( u 1 t , … , u M t , I t , ε y t ) .

The SCM formulation allows us to define (strong) interventions via the do-operator, disconnecting the intervened variable from its parents in the causal graph and assigning to it a specific value [35]. Thus, we quantify the impact of an intervention I on the target y at time t as

The first term is the observed, factual, post-intervention target, whereas the second term is the unobserved, counterfactual one. For SCSCMs, Zeitler et al. [18] showed that if causal mechanisms are invariant, and if the donors are proxies for the latents, then the counterfactual is identified via a unique function h of the (valid) donors such that E ( y t ∣ do ( I t = 0 ) , I t = 1 ) = E ( h ( x t , I t = 0 ) ) .

3.2 Spillover detection for selecting valid donors

The power of SC lies in the fact that the counterfactual in equation (1) can be identified even when there are post-intervention shifts in the exogenous errors P ( ε u t ) for the latents u . However, to identify the counterfactual we still need to specify donor units x that are valid. A donor is valid if it adheres to the DAG depicted in Figure 1a, and remains a proxy for the latents (Definition 3.2) at all time points. In particular, it must not be impacted by spillover effects from the intervention. Given an SCSCM from Definition 3.1, spillover effects manifest as post-intervention shifts in the donor errors P ( ε x t ) .

Usually, one must appeal to strong a priori domain knowledge in order to select valid donors. Can we leverage data from the donor pool itself, augmenting incomplete domain knowledge, to gain confidence that a potential donor is valid? Ideally, for a given donor candidate x i , we would like to be able to use pre-intervention data to test for shifts in P ( ε x i t ) that would rule out x i as a valid donor. Such a procedure would allow us to select donors based on a weaker form of domain knowledge than that required to know exactly which donors adhere to the DAG. This issue of selecting valid donors is distinct from the usual considerations of selecting donors based on pre-treatment fit [37], and is in fact a prerequisite.

In Section 3.2.1, we provide a theorem showing that the assumptions necessary for SC non-parametric identifiability – the proxy variable completeness condition (Definition 3.2), and invariant causal mechanisms (Definition 3.3) – also facilitate forecasting donor values given additional assumptions. We use this theorem to introduce a practical, theoretically grounded donor selection procedure based on detecting potential spillover effects to identify invalid donors.

3.2.1 Donor selection procedure: Theory

Theorem 3.1

If causal mechanisms are invariant, and the donors x 1 t − 1 , … , x N t − 1 are proxies for the latents u 1 t − 1 , … , u M t − 1 then, for each donor x i , there exists a unique function h i such that for all time points t we have:

(2) E ( x i t ) = E ( h i ( x 1 t − 1 , … , x N t − 1 , P ( ε x i t , ε u t ) ) )

The proof of this theorem can be found in Appendix A. Intuitively, because x i t is a function of the latents at time point t − 1 , and because the donors at t − 1 are proxies for those latents, we can swap the latents for the donors and write x i t as a function of x 1 t − 1 , … , x N t − 1 and the exogenous error terms at time t (Figure 1b).

We now use Theorem 3.1 to identify and exclude potentially invalid donors. Under conditions that (a) donor x i is valid, and (b) the exogenous error distributions P ( ε u t ) for the latents have not shifted at time t relative to their pre-intervention values, then Theorem 3.1 implies that we can forecast post-intervention values for x i based on pre-intervention donor data alone. Conversely, failing to forecast x i t implies the violation of condition (a), (b), or both. This observation forms the basis of our spillover detection method.

To use this theoretical result to obtain a practical method for flagging invalid donors, we assume that forecast errors are due to the donor unit error terms (due to being impacted by the intervention) and not the latent error terms. Crucially, we assume that failing to forecast x i t means that condition (a) is violated – that P ( ε x i t ) has shifted^[1] relative to pre-intervention values – and x i is not a valid donor.

In general, without additional domain knowledge, we cannot be sure that the unpredictability of x i is due to spillover effects and not changes in u j . However, in Section 3.4, we employ sensitivity analysis to formally bound the bias in our SC causal estimate violations to this assumption – i.e. false positives, when we exclude a donor x i even though it was the latent u j that changed, and false negatives, when we include a donor x i even though it was impacted by spillover effects. In particular, when our spillover detection incorrectly flags a donor x i (which is a false positive), then excluding this donor will only introduce bias if the remaining selected donors do not satisfy the completeness condition (i.e. there is omitted variable bias). This sensitivity analysis is the bridge that brings us from the theory of Theorem 3.1 to our practical donor selection method in Algorithm 1.

As we demonstrate in Section 4, violations of condition (b) – shifts in the latents – are only an issue for the selection method if they occur at the same time point as the donor forecast. If the latents shift later in the post-intervention period, this does not bias donor selection, as these later post-intervention data are not used in our selection procedure. Lags between the intervention and spillover effects can be dealt with by forecasting on coarsened, time-averaged donor data^[2]. Furthermore, time averaging can reduce false negatives in cases with very noisy donors (Figure 3). However, the longer time windows also increase the risk of false positives due to potential shifts in the latent distributions.

Figure 2

Bias τ ^ − τ and 95% Monte Carlo confidence intervals for 2000 simulated datasets. The data generating process is described in Appendix B. To construct SCs, we must identify the subset of donors that are valid. The horizontal axis shows the procedure used to identify this potentially valid set. For comparison, the case labelled All shows the expected bias of 1.6 when we assume that all donors are valid, and the case labelled Valid shows the bias when we have perfect knowledge about which donors are valid. Our S 1 and S 2 donor selection procedures are described in Section 3.2.1 and Algorithm 1. The standard deviation of the donor noise term, ε x t ∼ N ( 0 , σ ) , increases from left to right. As the noise increases, the spillover detection procedure is more likely to return false negatives, which increases the bias due to invalid donors. Note that even with optimal selection of valid donors, the estimates can still be biased due to donor noise (we address this residual bias in Figure 4). In the high noise case, despite being biased, our procedure still reduces bias relative to the case where we assume that all donors are valid. In this setup, 80% of the donors are invalid, and so our procedure has lower bias than the All case on average when fewer than 80% of the selected donors are invalid.

For the rest of the article, we restrict our attention to linear SC models, both because of their ubiquity, and also, more practically, to be able to discuss concrete sensitivity analysis bounds [18,19]. We make no distinction, however, between linear SCs as a consequence of the target and donors being linear functions of the latents, and linearity emerging non-parametrically due to the “aggregate unit” assumption of Shi et al. [21]. For demonstration purposes, we also focus on simple linear models for estimating h i as part of the donor selection procedure, although this is not a strict requirement (even when restricting to linear SC models, and linear sensitivity analysis). In principle, we can swap the linear model in this procedure for any flexible machine-learning model, using, e.g. conformal inference for constructing calibrated prediction intervals, perhaps improving performance (see [20,38], for applications of conformal inference in the context of SC models).

3.3 Using excluded donors to debias linear SC models

While the donors excluded by Algorithm 1 are not used in constructing the SC counterfactual, they can still be used to debias SC causal effect estimates, as we now show. The experiments in Section 4 use the following linear model for constructing SCs based on pre-intervention time points:

As discussed by Shi et al. [20], in situations where the donors are noisy (imperfect) proxies of the latents, the SC model in equation (4) might fail to estimate consistent donor weights β i , unless we assume perfect pre-treatment fit. Using proximal causal inference [20,24], we can leverage proxies not used in the construction of the SC to debias the estimates even with imperfect pre-treatment fit. In the proximal causal inference literature, this debiasing is typically accomplished with two-stage least squares estimation or the generalized method of moments. For the experiment in Figure 4, we opt to jointly model the target and donors as

where z l are the (potentially invalid) donors excluded by our selection procedure. Such a model can be readily estimated using a probabilistic programming language.

Figure 3

The left panel shows the bias in the high noise case of Figure 2. Our selection procedures S 1 and S 2 are severely biased due to invalid donors. The right panel shows where we time average the donor data, in buckets of five time points, before passing through the spillover detection. Note that we do not average pre- and post-intervention data in the same bucket, and we use the original, non-averaged data in the SC model. The averaging reduces false negatives such that the performance is on par with the optimal selection of valid donors. We address the residual bias in Figure 4.

How is it possible that we can use excluded donors z l , potentially impacted by spillover effects from the intervention, when constructing debiased SC estimates? Note that the excluded donors satisfy the proxy existence condition of Shi et al. [20], such that z l t ′ ⊥ ⊥ { y t ′ , x i t ′ } ∣ u j t ′ for all pre-intervention times t ′ . The key point allowing us to leverage potentially invalid donors is that we only ever use pre-intervention data from z l , and so the SC estimates are unaffected by their post-intervention dynamics.

To build intuition about the above procedure, consider the following model:

The variable Y is analogous to the target variable, U the latent, and X the donor (i.e. a noisy proxy of the latent U ). We would like to estimate the donor weight β . However, because U is latent, we cannot estimate the expectation E ( X ∣ U ) . If we regress Y on X we will not obtain a consistent estimate of β because the errors are correlated with X .

Instead, if we have an additional proxy variable Z , we can rewrite the model as

In this case, Z is observed and so we can estimate the expectation E ( X ∣ Z ) on the right hand side, and then obtain a consistent estimate for the donor weight β . One approach for estimation (often used for estimating instrumental variables) is a two-stage estimator where we first regress X on Z to estimate X ˆ (sometimes referred to as a proximal control variable), and then regress Y on X ˆ . The multivariate model in equation (5) effectively combines this two-stage process into a single model (similar models can be used for estimating instrumental variables, see, e.g., [39], Section 23.4).

Figure 4

The left panel shows the bias in the medium noise case of Figure 2. The estimated causal effects are biased even with optimal selection of valid donors. This is because the donors are noisy (imperfect) proxies of the latents, and so conditioning fails to fully close backdoor paths. As described in Section 3.3, we can debias these estimates using proximal causal inference to leverage donors excluded by the selection procedure. The right panel shows that the Valid and S 1 selection procedures now give unbiased estimates of the causal effect, although there is still some bias with procedure S 2 due to invalid donors. As in Figure 3, time averaging would reduce the bias in S 2 .

3.4 Sensitivity analysis

In this section, we discuss the false positive (excluding donors not impacted by spillover effects), and false negative (including donors impacted by spillover effects) failure modes of our donor selection procedure. We note that these failures can happen for multiple reasons including shifts in the latents at the same time as the intervention (as discussed in Section 3.2.1; see also Section 4.1 for an example), as well as misspecification and/or poor fit of the forecast model. Zeitler et al. [18] provided a general framework for sensitivity analysis in non-parametric SC models, bounding the bias when there are relevant latent variables with no observed donors as proxies (omitted variable bias). They also give a formula for calculating these bounds in the case of linear SC models, which we make use of in Sections 3.4.1, and 3.4.2. In the context of negative control, Miao et al. [19] introduced positive control outcome variables (which are similar to invalid donors) and discussed treating the spillover effect as a sensitivity parameter to investigate the plausibility of causal effect estimates. We make use of this approach in Section 3.4.3.

4.1 Simulated data

We construct 2,000 simulated datasets according to the data generating process described in Appendix B, with large pools of potential donors, most of which are impacted by spillover effects from the intervention. Each dataset consists of a target timeseries, 10 latents, and a pool of 1,000 potential donors. In each pool, a random set of 80 % of the donors are invalid, impacted by a spillover effect of − 2 .

For a given dataset, we must first attempt to identify the subset of valid donors. We refer to the set of donors returned by a selection procedure as the potentially valid donors (PVDs). Next, in order to simplify comparisons between different selection procedures, we estimate the SC using a sample of ten donors from the PVDs. This sampling step ensures that the resulting SC models all have the same number of parameters, regardless of the number of PVDs identified by the different selection procedures^[3]. In Appendix C, we compare this approach to one where we estimate SCs using the full set of PVDs, and employ regularization to enforce sparsity in the donor weights. For the data generating process considered here, the expected bias is the same with both approaches. We use the sampling approach to focus the comparisons on the selection of valid donors, rather than also having to consider possible differences due to selecting donors based on pre-treatment fit.

In Figure 2, we show the bias after applying our donor selection procedures S 1 ^[4] and S 2 . For comparison, the case labelled All shows the bias if we assume that all the donors are valid, and the case labelled Valid shows the optimal scenario if we had perfect knowledge about which donors are valid. Our selection procedures are very close to optimal in both the low and medium donor noise cases. However, the performance degrades as the donor noise becomes comparable in magnitude to the spillover effect. In Figure 3, we address this issue and show how time averaging improves performance when the donors are very noisy. In Figure 4, we leverage excluded donors to further debias effect estimates in situations with noisy donors, as discussed in Section 3.3. Finally, in Figure 5, we show how contemporaneous shifts in the latent distributions can bias our selection procedure, and that these latent shifts are not an issue if they occur later in the post-intervention period.

Figure 5

The left panel shows the bias when the error term for one of the latents shifts at the same time as the intervention. The error term for U 1 shifts from ε u 1 ∼ N ( 0,1 ) pre-intervention, to ε u 1 ∼ N ( 0.5,1 ) post-intervention. Our selection procedure incorrectly flags this as a spillover effect (false positive), thereby excluding donors that depend on U 1 . The right panel shows where this latent shift occurs just after the intervention. In this case our donor selection procedure recovers an unbiased estimate of the causal effect.

4.2 Semi-synthetic data

In this section, we further validate our donor selection procedure with using real-world data. In particular, we consider semi-synthetic variants of the 1990 reunification of West and East Germany [13], and the 25 cents tobacco tax increase in California in 1988 [12]. For each dataset, we introduce a semi-synthetic unit to the pool of potential donors, constructed to be a noisy proxy of the target as x syn t ∼ N ( y t , σ ) . Being predictive of the targets, these invalid donors receive large weights in the SC models and bias the effect estimates towards zero. This also highlights the critical distinction between selecting valid donors, and the usual considerations of selecting donors based on pre-treatment fit.

We assume SC models of the form y t ∼ N ∑ i β i x i t , σ y , β i ≥ 0 , ∑ i β i = 1 , β i ∼ Dirichlet ( 0.4 ) , and σ y ∼ N + ( 0,1 ) . The restriction to positive donor weights summing to 1 was used in the original models of Abadie et al. [12,13]. We further impose a Dirichlet ( 0.4 ) prior on β i which tends to regularize the weights towards sparse distributions, and we standardize the target and donors such each time-series has zero mean and unit standard deviation in the pre-intervention period. We construct 95% credible intervals by taking the 2.5 and 97.5% percentiles of samples from the posterior predictive distribution. We note that these uncertainty intervals are calculated after the donors have been selected and so do not account for additional uncertainty that the selection procedure introduces. Although beyond the scope of the current article (which focuses on identification of invalid donors), a promising future direction for this work would be to explore how uncertainty due to the donor selection procedure can be addressed using Bayesian selection-adjusted inference [40,41], or frequentist post-selection inference (see, [42], for a recent review).

In Figure 6, we show the estimates for the effect of German reunification on Germany’s per-capita GDP, and in Figure 6, the estimates for the effect of California’s tobacco tax increase on per-capita pack sales. The top panels show the effect estimates after applying our donor selection procedure. In this case, the semi-synthetic, invalid donors are correctly flagged and excluded from the SC models, resulting in causal effect estimates that are consistent with previous studies [12,13]. The SC models in the bottom panels include the semi-synthetic, invalid donors, which results in causal effect estimates much closer to zero than the original findings of previous studies [12,13].

Figure 6

(a) Estimated causal effect of German reunification. The intervention time is indicated by the vertical dotted line, and the shaded areas show the 95% posterior predictive intervals. The pool of 16 potential donors includes a semi-synthetic unit that is a noisy proxy of West Germany. In the top panel, we show the effect estimated after selecting ten donors using our S 1 selection procedure. The invalid donor is correctly flagged and excluded from the SC model. In this case, the results are very similar to Figure 3 from Abadie et al. [13], which is consistent with their findings about robustness to donor selection. In the bottom panel, we add the semi-synthetic unit to the SC model. This invalid donor receives a large weight, which biases the estimate towards zero. (b) Estimated causal effect of the California tobacco tax. The pool of 39 potential donors includes a semi-synthetic unit that is a noisy proxy of California. In the top panel, we show the effect estimated after selecting 30 donors using our S 1 selection procedure. The invalid donor is correctly excluded from the SC model, and the resulting effect estimate is very similar to Figure 3 from Abadie et al. [12]. Again, this is consistent with their findings about robustness to donor selection. In the bottom panel, we add the semi-synthetic unit to the SC model which biases the estimate towards zero. These results highlight the critical distinction between selecting valid donors, and the usual considerations of selecting donors based on pre-treatment fit.