Data-Adaptive Causal Effects and Superefficiency

1 Introduction

When causal effects are heterogeneous, then inferences depend on the population for which causal effects are estimated. Although population average causal effects have traditionally been the inferential targets, recent results have focused on estimating average causal effects that are local to some subpopulation for reasons of efficiency. These approaches include trimming observations based on the distribution of the propensity score [1], using regression adjustment to estimate reweighted causal effects [2, 3, 4], or implementing calipers for propensity-score matching [5, 6]. In some cases, the target parameter is dependent on the empirical distribution of the data, including cases where the researcher is explicitly conducting inference on, e. g., the average treatment effect among the treated conditional on the observed covariate distribution [7], or other causal sample functionals [8, 9], without revision to the estimator being used.

These approaches privilege efficiency in estimation over targeting population average causal effects, and often allow for the target to be defined on the basis of the observed data. We provide an example of how these approaches, taken to their extreme, can provide extraordinary gains in statistical certainty. We consider the case of a data-adaptive target parameter [10] that is allowed to vary with the data depending on which subpopulation’s local average causal effect is best estimated. When treatment effects are heterogeneous, adaptively changing the target parameter on the basis of efficiency yields an unusual result: if the population average causal effect can be consistently estimated with a root-n consistent and asymptotically normal estimator θˆ, then the same estimator θˆ is always superefficient (i. e., faster than root-n consistent) for a data-adaptive local average causal effect. Furthermore, with an additional regularity condition on mean square convergence, we show that the mean square error of θˆ for a data-adaptive local average causal effect is of o(n−1).

2 Results

Consider a full data probability distribution G with an associated causal effect distribution τ with finite expectation EG[τ], where EG[.] denotes the expectation over the distribution G. Further denote the support of the distribution of τ as SuppG[τ]. We impose a regularity condition on τ establishing non-degeneracy of τ.

Assumption 1 is equivalent to assuming that causal effects are not constant across observations in the distribution G; i. e., causal effects are heterogeneous.

We do not observe the full data probability distribution G, but we observe an empirical distribution Fn. Suppose that, using Fn, we have a root-n consistent and asymptotically normal estimator of the average causal effect EG[τ], θˆ.

We now define the target parameter, θFn.

The target parameter adapts naturally to the closest value in an interval surrounding EG[τ], where the width of the interval is defined by the support of τ. We formalize how each θFn is a local average treatment effect.

A proof of Proposition 1 follows directly from the fact that a weighted mean can obtain any value in the interval defined by the infimum and supremum of its distribution’s support. Proposition 1 asserts that across all realizations, the target parameter θFn corresponds to an average causal effect for at least one subpopulation. (There in fact may be infinitely many subpopulations to which θFn corresponds.) The composition of the subpopulation(s) associated with each θFn is not directly knowable by the researcher and may vary across realizations of the data.

However, mirroring results on other data-adaptive parameters under random sampling, including the sample average causal effect, the target parameter θFn will converge to the average causal effect EG[τ] at root-n rate. Proposition 2 proves that the data-adaptive local average causal effect is asymptotically equivalent to the average causal effect, and establishes its rate of convergence.

A proof of Proposition 2 follows by noting that n(θˆ−EG[τ])=Op(1) and that across every realization, |θFn−EG[τ]|≤|θˆ−EG[τ]|.

We now turn to our primary result, proving the superefficiency of θˆ in estimating θFn.

In short, Proposition 3 demonstrates that the probability that θˆ falls inside the support of the effect distribution converges to one quickly; conditional on this event, then estimation error is zero (as the target parameter takes on the value as the estimator with probability one). To illustrate this result, we can consider a case where an interval defined by the support of the effect distribution encompasses the sampling distribution of the estimator.

A proof of Corollary 1 follows by noting that Pr(|θˆ−EG[τ]|≤c)=0, and applying Definition 2. In other words, if the support of the estimator being used lies entirely within the interval [EG[τ]−c,EG[τ]+c], then estimation error is always zero. This condition necessarily holds if Supp_G[τ]=ℝ, then the value that any estimator θˆ takes must coincide with a local average causal effect. But note that Corollary 1 would not hold if Supp_G[τ]=ℝ⁺ and Pr(θˆ<0)>0.

Our results can be generalized to stronger claims straightforwardly. When a regularity condition is imposed on the rate of convergence of θˆ to normality, a stronger result can be obtained about the rate of mean square convergence.

3 Discussion

Our results highlight the additional certainty obtained by data-adaptively choosing the population for which average causal effects are measured on the basis of efficiency. It is well known that efficiency gains may be obtained through data-adaptive inference. But the extent to which the researcher can benefit from such practice has been understated. Under treatment effect heterogeneity – a precondition for locality to be a concern – all root-n consistent and asymptotically normal estimators of the average treatment effect are superefficient for a local average treatment effect.

There is of course a cost to this superefficiency: the target parameter is likely not of intrinsic interest. This issue is not unique to our setting, and other methods that change the inferential target based on efficiency concerns may be subject to this critique. As Crump et al. ([1], p. 188) notes, “external validity may be lost by changing the focus to average treatment effects for a subset of the original sample.” This is exacerbated in our setting by the researcher’s lack of knowledge about the characteristics of the subpopulation under study. Our result represents an extreme case of privileging efficiency over targeting population average causal effects. However, our results provide insight into a potential pathology of data-adaptivity purely on efficiency concerns: the gains in statistical certainty may be essentially unbounded without further restrictions. We hope that future work in the domain of efficiency theory for data-adaptive parameters will consider classes of restrictions that would exclude the case considered here.