A Fundamental Equivalence between Randomized Experiments and Observational Studies

1 Two study designs

Say you are charged with the task of learning if and to what extent a binary, time-fixed treatment, denoted as A with levels a=0,1, affects a binary outcome of interest, measured at some fixed time-point after treatment and denoted as Y with values y. Your goal is to produce an estimate of the expected difference in the outcome Y due to the treatment (e. g., risk difference (Cole et al. 2015). With limited resources, you must choose one of the two following study designs. You will want to choose the design that provides more information regarding this expected difference in the outcome Y due to the treatment. By “information” here we mean the width of the bounds for the risk difference.

The first study design is a randomized experiment (Fisher 1926). Specifically, you randomly assign n participants with proportion p to treatment A=1 and proportion 1−p to treatment A=0. Say a complication is that under this first study design exactly half of the outcome assessments will be missing, balanced equally across treatment groups. To emphasize, in this thought experiment, for illustrative purposes, half the outcome data are missing for the randomized experiment. The second study design is an observational (cohort) study. You observe n participants where (the same) proportion p choose treatment A=1 and proportion 1−p choose treatment A=0. Under this study design you observe all n outcome assessments.

Assume n is so large, and that the participants are a random sample from the target population, such that we can ignore the issues of random error and generalizability, respectively. Assume the outcome is measured without error in both studies. Assume that in the experiment there is complete compliance with assigned treatment, and in the observational study treatment is measured without error. Assume that there is no interference, i. e., one participant’s treatment does not affect the outcomes of another participant. Assume there are not different versions of treatment, or treatment variation irrelevance (Cole and Frangakis 2009; Vander Weele 2009). Finally, assume that each study is the same in terms of cost and other operational issues. Which study design would provide more information (i. e., tighter bounds)?

2 Fundamental equivalence

Without additional context, both study designs presented provide the exact same amount of information (i. e., width of bounds) about the average difference in Y due to treatment! To see this point clearly, define hypothetical values of Y under each treatment, which are called potential outcomes (Holland 1986; Neyman et al. 1990; Robins 1986; Rubin 1974). The difference in Y due to treatment is the difference between the value of Y if a participant had been assigned (or chosen) treatment A=1, denoted Y1, and the value of Y if a participant had been assigned (or chosen) treatment A=0, denoted Y0. Define the expected difference in the outcome Y in the target population due to the treatment by EY1−Y0. A fundamental problem of causal inference is that we never observe both Y1 and Y0 for the same participant. If we did observe Y1 and Y0 for each participant, then the average of Y1−Y0 would be an unbiased estimate of EY1−Y0.

For both studies, when we observe Y for a participant with A=a, then Ya=Y by counterfactual consistency (Pearl 2010), under our assumptions of no exposure measurement error and no versions of treatments, or treatment-version irrelevance (Cole and Frangakis 2009; Vander Weele 2009). Imagine for a moment that the first (experimental) study design was implemented with no missing data. Then counterfactual consistency allows us to observe 1/2 of the potential outcomes, namely Ya for A=a. The average of these observed outcomes for individuals assigned A=1 is an unbiased estimator of EY1|A=1, which equals EY1 because treatment assignment is independent of the potential outcomes by randomization. Similarly the average of the observed outcomes for individuals assigned A=0 is an unbiased estimator of EY0|A=0, which by randomization equals EY0. Thus the difference in average outcomes between the two arms of the study is an unbiased estimator of the treatment effect EY1−EY0=EY1−Y0. This is a central reason randomized designs are so highly regarded. Unfortunately, however, for the randomized experiment under consideration, half of the outcomes assessments are missing. The mechanism by which outcome assessments are missing is unknown.

In the second (observational) study design, counterfactual consistency again allows us to observe 1/2 of the potential outcomes, namely Ya for A=a. But we do not observe the other half of the potential outcomes, namely Ya for A≠a. Because the treatment was not randomized, the mechanism by which individuals chose treatment is unknown.

Because potential outcomes are missing due to an unknown mechanism under either design we cannot with certainty obtain an unbiased estimator of the expected difference in the outcome Y due to treatment, EY1−Y0. However, with the observed data, we can calculate bounds on EY1−Y0 under each design. These bounds are the same for each of the two study designs, proving the studies provide the same information about the effect of treatment. A mathematical proof of the equivalence of the bounds is provided in the Appendix.

3 An example

Table 1 provides a simple worked numerical example. In the upper panel of Table 1 the experimental study is represented. There are n participants, with 0.5n assigned to A=1 and 0.5n assigned to A=0. Among the 0.5n with observed outcome status, there are 0.025n treated events (i. e., A=1,Y=1) and 0.05n untreated events (i. e., A=0,Y=1), yielding a risk difference of – 0.1. Accounting for the 0.5n participants with missing outcomes, the logically possible range (i. e., bounds) of the risk difference are – 0.55 and 0.45. Specifically, the lower bound occurs if we assume all participants with missing outcomes assigned to A=1 are non-events (i. e., Y=0) and all participants with missing outcomes assigned to A=0 are events, or 0.025/0.5−0.3/0.5=0.05−0.6=−0.55. The upper bound occurs if we assume all treated participants with missing outcomes are events and untreated participants with missing outcomes are non-events, or 0.275/0.5−0.05/0.5=0.55−0.1=0.45.

In the lower panel of Table 1 the observational (cohort) study is represented. Again, there are n participants, with 0.5n treated A=1 and 0.5n untreated A=0. Here every factual outcome Y is observed but the counterfactual outcomes Ya for A≠a are missing. Accounting for the n missing potential outcomes, the bounds of the risk difference are again – 0.55 and 0.45. The lower bound occurs if we assume all untreated participants would not have experienced an event if, contrary to fact, they had been treated, and all treated participants would have experienced the event if, contrary to fact, they been untreated. In this scenario, the 0.05n events observed among participants with A=1 are the only events that would have been observed had all participants been treated (i. e., PY1=1=0.05) and the 0.1n events observed among participants with A=0 would be added to the 0.5n events we would have observed among participants with A=1, had they been untreated (i. e., PY0=1=0.6), for a risk difference of 0.05−0.6=−0.55.

The upper bound occurs if we assume all untreated participants would have experienced an event if, contrary to fact, they had been treated, and all treated participants would have not experienced the event if, contrary to fact, they had been untreated. Under this scenario, we assume that all 0.5n untreated participants would have had the event if (contrary to fact) they had been treated in addition to the 0.05n events we observed among participants with A=1 participants, so PY1=1=0.55. Similarly, we observed 0.1n events among participants with A=0 and assume no treated participants would have had events if (contrary to fact) they had been untreated, so PY0=1=0.1, making the upper bound of the risk difference 0.55−0.1=0.45.

4 Discussion

With a pair of treatments and assuming no additional context beyond what is provided above, the bounds for the risk difference from an observational study are equivalent to the analogous bounds for a randomized experiment with 50 % missing outcomes. Of course the described experiment may be labeled as “broken” because of the missing outcomes (Frangakis and Rubin 1999; Little et al. 2012). However, experience suggests that all real-world experiments are broken to varying degrees. In real-world settings each successive additional piece of context will unbalance the equivalence given in this example, giving preference to one design over the other. A central point of this paper is that we cannot conclude one design is better than the other without additional context. Such additional context, could be used to “unbalance” the equivalence and allow for an informed design choice. Yet identifying this balancing point, or equivalence, between a randomized experiment with missing outcome assessments and an observational study is useful. When a randomized experiment is feasible, this equivalence indicates that an experiment will be preferable to an observational study provided less than 50 % of outcome assessments in the experiment are missing. However, experiments are sometimes unethical and are often prohibitively expensive. When a randomized experiment is infeasible, nonexperimental observational studies can help to refine our knowledge, or sharpen our (probabilistic) bounds about the effect of a treatment. Moreover, balancing points, such as described here, provide a natural foundation when considering both the design and interpretation of experimental and nonexperimental studies.