Temporal discontinuity trials and randomization: success rates versus design strength

Appendix A: Mathematical proofs

Define

and restate the optimization problem in (1), without the optional constraint, and only the U objective function, as

In (14) the variables are (s, t, u, v), and in (14a) the objective function to be maximized is (s + u + v) while the objective function to be minimized is t, and the constraints are described on lines (14b), (14c), (14d), and (14e). By adding (14d) and (14e) we obtain

Subtracting (15) from (14c) results in

Reasoning from (16) and (14b), to minimize t we push s to its minimum and set

resulting in

From there via (14a) we obtain

By symmetry we obtain

Denote the proportion of the population that is affected by the treatment with

From (14e) and (14d) we have

From (14c) and (17) we obtain

If α is known then the equation in (19) is an optional constraint for the problem described in (14). Adding (16) and (19) results in

from which we obtain

Likewise,

Given (r _j , r _k ) the function U(α) linearly maps its domain [|r _j  − r _k |, min{r _j  + r _k , 1}] (see (18)) onto a space of success rates ranging from max{r _j , r _k } up to min{r _j  + r _k , (r _j  + r _k  + 1)/2, 1}=min{r _j  + r _k , 1}, and the function L(α) linearly maps its similar domain [|(1 − r _j ) − (1 − r _k )|, min{(1 − r _j ) + (1 − r _k ), 1}] onto a space of success rates ranging from 1 − max{1 − r _j , 1 − r _k } down to 1 − min{(1 − r _j ) + (1 − r _k ), ((1 − r _j ) + (1 − r _k ) + 1)/2, 1}=1 − min{(1 − r _j ) + (1 − r _k ), 1}.

Appendix B: Discontinuity designs

Causal inference from observational data is a challenging problem. Natural experiments are one way to approach the problem. In particular, the methodology of Eckles et al. [30] is ideal, assuming an infinite population. They describe noise-induced randomization with a discontinuity design. There is a measured running variable Z=U + ϵ, where U is the real latent variable and ϵ is measurement noise. Treatment is assigned to those individuals with Z≥c for some cutoff value c and withheld from those individuals with Z<c. For small δ>0 we may restrict our attention to the sub population of individuals with Z ∈ (c − δ, c + δ). In the limit as δ → 0 treatment assignment is haphazard and in most circumstances it is therefore reasonable to conclude that all covariates are balanced and that causal inference is warranted.

An example will provide context and clarity. We adapt the methodology and notation of Eckles et al. [30] and apply it to the similar study of low birth weight infants conducted by Almond et al. [41]. The threshold of very low birth weight is c=1500 g. Infants born below the threshold are treated with intensive care, while infants born above the threshold are treated with standard care. It has been observed that infants born with birth weights just above 1,500 g have an infant mortality rate of approximately 5.5 %, while infants born with birth weights just below 1,500 g have an infant mortality rate of approximately 4.5 %[41]. Ideally, we have Z=U + ϵ where Z is the measured birth weight, U is the actual birth weight, and ϵ is exogenous measurement error. The latent variable U is a potential confounder. However, provided with a large enough population, on the sub population of infants with Z ∈ (c − δ, c + δ), in the limit as δ → 0, it may be reasonable to conclude that each strata of U will be equally balanced above and below c. With exogenous noise ϵ and treatment actively assigned from Z<c where Z=U + ϵ, we conclude that balance of U implies balance of all unmeasured covariates thus warranting causal inference from this noise-induced natural experiment.

It is worth discussing a subtle point relating to that warrant for causal inference from a natural experiment. If a single treatment were not actively assigned by an experimenter with control over the situation then we would not be able to conclude that all unmeasured covariates were balanced. Suppose for instance in the context of a natural, natural experiment [42] nature assigns multiple exposures. Consider as the treatment the primary exposure, but note that there is no way to know whether nature balanced the secondary exposures or not. This subtle point is discussed in further detail in ([37]; Section 5.2). It is related to the stable unit treatment value assumption ([43]; p. 10) and also the exclusion restriction ([44]; Section 3.2). Similar concerns about compound treatments arise in randomized experiments [45].

The noise-induced discontinuity design is a clever approach, but its results are not as compelling as the results of an actual experiment conducted on a representative sample of a well defined population. The problem becomes apparent when it is realized that the noise-induced discontinuity design requires conditioning on the results of a noisy process. Recall how Z=U + ϵ. Suppose continuous probability density functions h and g for U and ϵ, respectively. The probability density function h represents the population distribution of U values. The probability function g represents exogenous measurement error, assumed to be independent and identically distributed across individuals. The sub population of individuals with Z=c have a U distribution with a density function proportional to h(u)g(u − c). If we assume relatively small variance of the noise then h(u) is roughly constant near c and conditional on Z=c we have a distribution of U values with probability density function approximately equal to g(u − c). For instance, if the measurement noise is Gaussian then we have a warrant for causal inference on a sub population of individuals with U values normally distributed about the cutoff c. But that is not a representative sample of any sub population well defined from U. If we allow heterogeneous causal effects that depend on U then our causal inference could be biased unless the noise term is uniformly distributed.

This is a subtle point worth restating in the language of our example involving low birth weight infants. Yes, we can conduct causal inference; we can conclude that the treatment of intensive care provided to very low birth weight infants saved lives. No, we can not establish an intervention policy on a well defined sub population and estimate the average effect of our intervention, if the noise distribution is anything but uniform. We can’t say for example that intensive care saves approximately 1 % of infants with actual birth weights between 1,499 and 1,501 g, because there are alternate explanations on that particular sub population. It could be for example with Gaussian noise that the treatment of intensive care saved lives on the over sampled segment right near the cutoff of c=1500 but lost lives on the under sampled segments slightly further away from the cutoff near c=1499 g and c=1501 g. That scenario could be consistent with both a) the observed data that arose from a non-representative sample conditional on Z which contains noise and b) a smaller or non-existent average treatment effect on that sub population with 1499<u<1501, depending on the variance of the noise; see Figure 1. While a continuity or monotonicity of effects argument could be made, the thought experiment just described in this section reveals the importance of a well-defined population for intervention policy.

Figure 1:

With Gaussian noise we observe a Gaussian distribution of U values conditional on Z=c=1500 which could result in an over selection of individuals who benefit from treatment (indicated with plus symbols in the plot) and an under selection of individuals who are harmed by treatment (indicated with minus symbols in the plot), warranting causal inference on the sub population with 1499<u<1501 while the average treatment effect on that same sub population is zero, limiting the effectiveness of any policy recommendation for the treatment across the whole sub population.

Appendix C: Conditional interventions

Pearl [46] represents interventions with the “do operator”. Within that framework an intervention is a “hypothetical situation in which treatment … is administered uniformly to the population” [46]. That definition of an intervention means that all individuals of a population have their treatment variable value set to the same prescribed value. Those individuals within the population whose treatment variable status was already at that prescribed value remain unaffected by such an intervention.

However, commenting on the use of causal diagrams for empirical research, Imbens and Rubin [47] have written the following.

Important subject-matter information is not conveniently represented by conditional independence in models with independent and identically distributed random variables. Suppose that, when a person’s health status is ’good’, there is no effect of a treatment on a final health outcome, but, when a person’s health status is ’sick’, there is an effect of this treatment, so there is dependence of final health status on treatment received conditional on initial health status. Although the available information is clearly relevant for the analysis, its incorporation, although immediate using potential outcomes, is not straightforward using graphical models.

There are graphical models that unify causal graphs and potential outcomes 48], [49], [50], [51, but here we will utilize potential outcomes notation.

Merriam-Webster’s dictionary defines a medical intervention as an “act of interfering with the outcome or course especially of a condition or process (as to prevent harm or improve functioning)” [52]. We emphasize that an intervention is an act of interfering, and we proceed to define conditional interventions as follows, so as to preclude consideration of the contradictory idea of intervening on individuals who do what they already planned to do.

Consider those individuals who plan on treatment X=x. An intervention conditional on x is the act of interfering and assigning X=(x + k), for some k≠0, to each individual who plans on treatment X=x. This is the same as Pearl’s definition previously described but applied only to individuals with X=x. However, we conceptualize this conditional intervention as doing the change k rather than doing the changed treatment value x + k. Conditional interventions can be applied to sub populations who would benefit, and the notation we just introduced helps to describe a planned intervention on individuals at various and specific times. When an individual, e.g. a person, is repeatedly measured at different points in time, it is recommended to consider the measurements as arising from different experimental units [43]. The simplest case of a conditional intervention arises in a cross sectional study with a binary treatment indicator X=0 for control and X=1 for treatment. In that simplest case we may apply an intervention of k=1 to those who prefer control and an intervention of k=−1 to those who prefer the treatment.

Suppose that X causes Y and we have observed the data of Table 1. The causal definition of no-confounding is P(y|do(x))=P(y|x) [53]. In potential outcomes notation, for x=2, the causal definition of no-confounding asserts that E(Y _x=2)=E(Y|x=2), which is 1/2 in Table 1. The causal definition of no-confounding is satisfied. However, we could have E(Y _x=2|x=0)=0 and E(Y _x=2|x=1)=1, so the causal definition of no-confounding is technically still satisfied, but there is clearly causality apparent in the conditional, potential outcomes, and no association in the observed data. This simple example is related to what has been described as unfaithfulness of a population to a causal graph ([54]; Section 4). To avoid the assumption of faithfulness it is therefore reasonable to introduce a causal definition of conditional no-confounding to pair with our definition of conditional interventions. Basically, conditional no-confounding means E(Y _x+k |x)=E(Y|x + k). We can condition on more than just the treatment variable X, as theoretically considered implicitly in the formulation of the constrained optimization problem in (1).