1 Background

Causal effects often differ between groups of people. Consequently, investigators are often required to reason carefully about which measures of effect, if any, can be expected to remain homogeneous between different populations or different subgroups. Investigator beliefs about effect homogeneity have important implications both for model specification, and for the choice of summary metric in meta-analysis (Deeks and Altman 2008). It is commonly believed that the risk ratio is a more homogeneous effect measure than the risk difference, but recent methodological discussion has questioned the evidence for the conventional wisdom (Poole, Shrier, and VanderWeele 2015; Panagiotou and Trikalinos 2015).

Approaches to effect homogeneity based on standard effect measures have several noteworthy shortcomings, summarized for reference in Table 1:

1. Approaches that assume equality of the risk ratio or the risk difference may make predictions outside the bounds of valid probabilities.

2. The odds ratio and the risk ratio have “zero-constraints": If the baseline risk in a population is either 0 or 1, approaches based on assuming that the targeted odds ratio is equal to the odds ratio in a referent population will necessarily result in concluding that the exposure has no effect (Deeks 2002).

3. If the risk difference, risk ratio, or odds ratio are equal across different populations, then the proportion that responds to treatment in a given population is required to be a function of that population’s baseline risk (Schechtman 2002).

4. The odds ratio is not collapsible; i.e the marginal value of the odds ratio may not be equal to a weighted average of the stratum-specific odds ratios under any weighting scheme, even in the absence of confounding and other forms of structural bias (Huitfeldt, Stensrud, and Suzuki 2016).

5. The predictions of the risk ratio model are not symmetric to whether the parameter is based on the probability of having the event, or the probability of not having the event, i.e whether we “count the living or the dead” (Sheps 1958).

In addition to the five points discussed above, we note that no generally applicable biological mechanism has been proposed that would guarantee the risk ratio, the risk difference, or the odds ratio stays constant between different populations. For earlier discussion of how biological mechanisms relate to the choice of effect measure, we point the reader to Siemiatycki and Thomas (1981) and Thompson (1991).

In this paper, we are interested in the effect of binary treatment A (e.g. a drug) on binary outcome Y (e.g. a side effect) in two separate populations P = s and P = t. In all examples, we have randomized trial evidence for the average causal effect of the treatment in study population P = s , and wish to predict the effect of introducing the treatment in the target population P = t, in which the drug is not available and in which we can only collect observational data. Counterfactuals will be denoted using superscripts. For example, Ya=0 is an indicator for whether an individual would have experienced side effect Y if, possibly contrary to fact, she did not initiate treatment with drug A. Because we have a randomized trial in study population s, the average baseline risk Pr(Ya=0=1|P=s), and the average risk under treatment, Pr(Ya=1=1|P=s) are identified from the data. Since treatment is not available in target population t, the average baseline risk, Pr(Ya=0=1|P=t), is also identified from the data. Our goal is to use this information, in combination with some plausible assumption about effect homogeneity, to predict the average risk under treatment in the target population, Pr(Ya=1=1|P=t).^[1]

Briefly, we recall that VanderWeele (2012) defined two separate types of effect heterogeneity: Effect modification in distribution (where the distributions of counterfactual variables vary across populations) and effect modification in measure (where a particular effect measure varies across populations). Here, we will propose a novel approach to effect homogeneity based on a new class of non-identified effect parameters; and show how this framework can sometimes be used to reason about homogeneity in terms of standard, identifiable measures of effect such as the risk difference and the risk ratio. Specifically, we will consider two risk ratios, which are equivalent to the two risk ratios considered by Deeks (2002), and defined as follows:

This paper is organized as follows. In part 2, we describe a scenario to illustrate the motivation for expanding upon existing effect parameters. In part 3, we introduce counterfactual outcome state transition (COST) parameters, and propose a definition of effect homogeneity based on these parameters. In part 4, we discuss the conditions under which COST parameters are identified from the data, and the implications of violations of these conditions. In part 5, we show that the COST parameters are not symmetric to the coding of the exposure variable, and discuss the implications of this observation. In part 6, we link COST parameters to substantive knowledge by providing examples of biological processes that result in effect homogeneity. In part 7, we discuss the empirical implications of our model. We conclude in part 8.

2 Motivating example

Suppose a team of investigators have data from a randomized trial where the risk of a particular side effect was 2% under no treatment and 3% under treatment. They wish to predict the effect in a different target population, where the baseline risk of the side effect is 10% (Table 2). For simplicity, we postulate that this treatment does not prevent the outcome from occurring in any individual (“monotonic effect”); this assumption will be discussed in detail later in the paper.

While discussing their data analysis plan, the first investigator postulates that the standard risk ratio RR(−) is equal between these two populations. Under this assumption, he estimates that 15% of the target population will experience the side effect under treatment. However, a second investigator believes that RR(+), not RR(−), might be equal between the two populations. Under this assumption, he estimates that 10.9% of the target population will experience the side effect under treatment.

A third investigator notes that neither the first nor second investigator have substantive arguments for choosing the RR(+) versus the RR(−). However, she realizes that the group at risk of being adversely affected by treatment is the 90% of the target population who were originally destined not to experience the side effect. She points out that the first investigator’s assumption (i.e. that RR(−) was equal between the populations) results in a prediction that among people who are at risk of being adversely affected by the treatment, the proportion who respond is much higher in population t than in population s, simply because the baseline risk is higher.

Given this, the third investigator instead suggests assuming a specific probability is constant across the populations: the probability that a person who was previously not destined to experience the outcome, does not experience the outcome in response to treatment. This probability can be computed in the trial as 97%98%≈99% and applied to the 90% who were originally destined to be unaffected is the target population. In this case, the third investigator’s approach results in the exact same estimate as the second investigator’s approach: an estimated 10.9% of the target population will experience the side effect under treatment.

Variations of the third investigator’s arguments have arisen independently multiple times in the literature, dating back more than half a century (Sheps 1958; Deeks 2002), but these recommendations have rarely been translated into common practice. Throughout the rest of this paper, we will formalize this line of reasoning in a counterfactual causal model, in order to explore its scope, limits and implications.

3 Definition of the counterfactual outcome state transition parameters

We define “counterfactual outcome state transition” (COST) parameters based on the probability that a person who becomes a case if untreated remains a case if treated, and by the probability that a person who does not become a case if untreated remains a non-case if treated. We also offer interpretations of these quantities in terms of the four deterministic response types (Greenland and Robins 1986) (Table 3). A list of parameters considered in this paper is shown in Table 4.

In a deterministic model, G can be interpreted as the proportion who are “Doomed”, among those who are either “Doomed” or “Preventative” and H can be interpreted the proportion who are “Immune”, among those who are either “Immune” or “Causal”. We will refer to G and H as the COST parameters for introducing treatment. G and H can be indexed for a specific population with subscripts (e.g. Gs is the parameter G in population s). If the COST parameters for introducing treatment are equal between two populations (i.e. if Gt=Gs and Ht=Hs,) this can equivalently be written as

Similar conditions were considered in different contexts by Gechter (2015) and by Athey and Imbens (2006). In our setting of binary outcomes, this definition of effect homogeneity addresses all previously discussed limitations of the standard effect measures: The underlying parameters have no baseline risk dependence, do not produce logically impossible results, are collapsible over arbitrary baseline covariates, and have no zero constraints. Further, if the coding of the outcome is altered, the only consequence is that the values of G and H are reversed. (See Appendix A)

4 Identification of the counterfactual outcome state transition parameters

Now that we have defined COST parameters, and described their motivation and attractiveness, we proceed to discuss how we can compute them in our studies. In an ideal randomized trial, we can identify the standard effect measures (RR, RD) without further assumptions beyond those expected to hold by design. Unfortunately, this is not the case for COST parameters: COST parameters are generally not identifiable without further assumptions. Therefore, even if we (somehow) knew that the parameters G and H are equal between two populations, we may not be able to use this fact to predict what happens when we introduce treatment in the target population. We next introduce assumptions that lead to identifiability (Propositions 1, 4), and discuss scientific and clinical implications when we are (Propositions 2, 5) and when we are not (Propositions 3, 6) willing to make those assumptions. The key identifiability assumption is monotonicity (VanderWeele and Robins 2010, 2009): We say there is non-increasing monotonicity if individuals who do not get the outcome if untreated, do not get the outcome if treated: Pr(Ya=1=1|Ya=0=0)=0. In other words, H = 1. Similarly, non-decreasing monotonicity occurs if individuals who get the outcome if untreated also get the outcome if treated, in which case G = 1.

Monotonicity is defined in the context of the specific exposure-outcome under consideration, and its plausibility therefore needs to be evaluated on a case-by-case basis. This means that within a trial, we may need to consider the plausibility of monotonicity for each outcome of interest separately. For instance, the antiarrhythmic agent Amiodarone can cause arrhythmia in some individuals and prevent it in others, and therefore does not have monotonic effects for the arrhythmia outcome. However, if the outcome of interest is a side effect such as pulmonary fibrosis, monotonicity may be a viable assumption because the drug is unlikely to prevent any individual from getting pulmonary fibrosis. In general, monotonicity could be a more reasonable approximation in situations where the outcome is a side effect strongly associated with pharmacological treatment.

The following three propositions are exactly symmetric to the preceding results, but apply to situations where exposure increases the risk of the outcome (i.e. monotonicity holds in the other direction). In such situations, we identify H instead of G:

Proofs of propositions (1–6) are provided in Appendix B.

5 Asymmetry to the coding of the exposure variable

So far, we have focused on problems – and resolutions to some problems – related to how the investigator encodes the outcome variable in the database. However, COST parameters are not invariant to the coding of the exposure variable. To illustrate, reconsider the trial where the risk under treatment was 3% and the risk under no treatment was 2%. If we reverse the coding of the exposure variable, we notice that the new exposure variable in fact reduces the risk of the outcome, meaning that a naive application of our approach would suggest using the standard (but now inverted) risk ratio 1RR(−) to estimate the probability of being unaffected by treatment. However, there is a subtle but important distinction from the earlier approach: by changing the definition of exposure, we have also changed the meaning of the parameter so that it is now defined as the probability of being unaffected by treatment among those who would have become cases under treatment, rather than among those who would have become cases under no treatment. This can be conceptualized as the effect of removing treatment from a fully treated population.

We will refer to the COST parameters associated with removing treatment as I and J. The choice of coding of the exposure variable is then equivalent to choosing whether to model equality of effect based on the parameters G and H, or based on the parameters I and J. For notational simplicity, we will continue to use the original coding of the exposure variable, and instead frame the question in terms of whether an investigator should define equality of effects based on the parameters G and H, or the parameters I and J.

In a deterministic model, I can be interpreted as the proportion who are “Doomed”, among those who are either “Doomed” or “Causal”, and J can be interpreted the proportion who are “Immune”, among those who are either “Immune” or “Preventative”. If the COST parameters for removing treatment are equal between two populations (i.e. if It=Is and Jt=Js,) this can equivalently be written as Ya=0⊥⊥P|Ya=1. With these definitions, results exactly analogous to propositions 1 through 6 can be derived for I and J.

As implied earlier in this section, one can construct examples to show that equality of COST parameters for introducing treatment does not imply equality of COST parameters removing treatment. In fact, if the baseline risks differ, then the two homogeneity conditions rarely hold simultaneously (an obvious exception to this would be under the sharp causal null hypothesis). This observation arguably presents a major conceptual challenge to the claim that our definition captures the intuitive idea of “equal effects”: to make our inferences invariant to the coding of the outcome, we have made them dependent on the coding of the exposure. However, in the next section, we show that, with some background knowledge about biological mechanisms, it is possible to reason about whether the COST parameters for introducing treatment are more likely to be homogeneous than the COST parameters for removing treatment. In particular, we show that it is possible to provide models for the data-generating process that guarantee equality of the COST parameters for introducing treatment, equality of the COST parameters for removing treatment, or neither.

6 Biological knowledge and equality of treatment effects

While it is usually not possible to reason a priori about whether the risk difference or odds ratio are equal between populations, we now provide a simple example to show that it is sometimes possible to reason based on biological knowledge about whether the COST parameters are equal between populations. This is intended only as a proof-of-concept, and the example is purposefully oversimplified in order to illustrate the principles. An outline of a formal treatment of these ideas is provided in Appendix C.

Consider a team of investigators who are interested in the effect of antibiotic treatment on mortality in patients with a specific bacterial infection. Since this antibiotic is known to reduce mortality, the investigators need to decide whether to report RR(−) as an approximation of G, or alternatively RR(+) as an approximation of 1J. In order to ensure external validity, this choice will be determined by their beliefs about which parameter is most likely to be constant across populations.

The investigators believe that the response to this antibiotic is completely determined by an unmeasured bacterial gene, such that only those who are infected with a bacterial strain with this gene respond to treatment. The prevalence of this bacterial gene is equal between populations, because the populations share the same bacterial ecosystem. If, as seems likely, the investigators further believe that the gene for susceptibility reduces mortality in the presence of antibiotics, but has no effect in the absence of antibiotics, they will conclude that G may be equal between populations. If, on the other hand, they had concluded that the gene for susceptibility causes mortality in the absence of antibiotics but has no effect in the presence of antibiotics, they would instead expect equality of J across populations.

For many antimicrobial therapies, the microbial genes that determine antibiotic susceptibility generally have functions that are perhaps better approximated by the first approach, and therefore motivates the choice to model the data as if G is equal between the populations. In other situations, in the presence of different subject matter knowledge, similar logic could be used to reach different conclusions. One example of this occurs in pharmacological applications involving adverse reactions to drugs, where it may be reasonable to use the parameter H if the determinants of susceptibility are equally distributed between populations, under certain assumptions about how those determinants interact with the drug. In many realistic applications, it may be more plausible that the COST parameters for introducing treatment are equal than the corresponding parameters for removing treatment, but this is by no means universal: for example, if some humans had retained our ancestors’ ability to synthesize Vitamin C endogenously, then the effect of fresh fruit on scurvy might be better modeled based on the parameter J.

In most settings, the particular function of the attribute that determines treatment susceptibility will not be known. In such situations, it is necessary to reason on theoretical grounds about which model is a better approximation of reality. One possible approach to determine whether either type of effect equality is biologically plausible would be to consider how an attribute or gene with the necessary function avoided either reaching fixation or being selected out of existence. For example, a genotype that causes the outcome in the presence of exposure will very likely go extinct in a population where everyone is exposed, but may survive in a proportion of a population where everyone is generally unexposed. This equilibrium may be equal between different groups of people (for example, equal between men and women in the same gene pool). Therefore, if treatment was unavailable in recent evolutionary history, the COST parameters for introducing introducing treatment may be more likely to be equal than the COST parameters for removing treatment. Similarly, an attribute that prevents the outcome in the absence of exposure will quickly reach fixation if everyone is unexposed, but its absence may survive in a small, stable fraction of the population if everyone is exposed. Therefore, if everyone were exposed in recent evolutionary history, the COST parameters for removing treatment may be more likely to be equal than the COST parameters for introducing treatment. Thus, this line of reasoning may provide an additional viable argument for choosing the index level of the exposure variable based on the value which it took by default in recent evolutionary history.

7 Testing for heterogeneity

If we believe that the COST parameters for introducing treatment are equal across populations, an empirical implication is that meta-analysis based on RR(−) will be less heterogeneous for exposures which monotonically reduce the incidence of the outcome, whereas meta-analysis based on RR(+) will be less heterogeneous for exposures which monotonically increase the incidence of the outcome.

However, this observation is complicated by an additional asymmetry associated with the ratio scale: If the outcome is rare (which is usually the case), Pr(Ya=1=0) and Pr(Ya=0=0) will both be close to 1, and RR(+) will therefore also be close to 1, even if treatment has a substantial effect. This results in a compression of the RR(+) scale, such that when heterogeneity is measured in terms of the absolute differences between effect sizes, clinically meaningful heterogeneity between populations will only be apparent at the second or even third decimal space. In contrast, heterogeneity on the RR(−) scale generally manifests itself at the first decimal. Therefore, any attempt to measure heterogeneity based on the absolute difference between each study’s estimate and the overall meta-analytic estimate will result in higher values for RR(−) than RR(+) for rare outcomes, for reasons that arguably say more about the mathematical differences between the scales than about their relative usefulness for summarizing effect sizes.

One option may be to quantify each study’s deviation from the common effect in terms of the lowest possible proportion of individuals whose outcome variable must be “switched" in order for the effect estimate in the study to equal the overall meta-analytic estimate. Another potential approach to this problem is to use the Risk Difference (RD) in place of RR(+) for exposures which increase the incidence of Y. In Appendix D, we show that if the outcome is rare, effect homogeneity on the RR(+) scale implies near-homogeneity on the risk difference scale. This approach is consistent with previous suggestions to consider the “relative benefits and absolute harms" (Glasziou and Irwig 1995) of medical interventions. Investigators using this approach must further keep in mind the potential differences in power between tests for homogeneity on the additive and multiplicative scales (Poole, Shrier, and VanderWeele 2015).

8 Conclusion

We have proposed a new approach to considering effect equality across populations, which avoids several well-established shortcomings of definitions based on standard effect measures. Our approach distinguishes equality of the effect of introducing the treatment to a fully untreated population from equality of the effect of removing the treatment from a fully treated population; and therefore requires investigators to reason carefully about the distinction between the two homogeneity conditions. Further, we provided examples of biological models that correspond to each form of effect equality. While the utility of our approach is limited to the restricted range of applications where these biological models are a reasonable approximation of reality, we believe such applications could occur with some frequency when studying the effectiveness and safety of pharmaceuticals.

If investigators are willing to assume that the COST parameters for introducing treatment are equal between populations, it follows that the standard risk ratio RR(−) should be used for exposures which monotonically reduce the incidence of the outcome, and that the recoded risk ratio RR(+) should be used for exposures which monotonically increase the risk of the outcome. Thus, the risk ratio will generally be constrained between 0 and 1. If the outcome is rare, RD may be used in the place of RR(+) for exposures that increase the incidence of the outcome. If the effect of exposure is not monotonic, the investigator may still choose the risk ratio model based on whether treatment increases or decreases the risk on average; in such situations, the extent of bias will be small if the extent of non-monotonicity is small, or if the populations have comparable baseline risks. This approximation is highly sensitive to violations of these conditions, and if there is reason to suspect substantial non-monotonicity, identification is not feasible and investigators may consider using an alternative approach to effect homogeneity (Bareinboim and Pearl 2013; Cole and Stuart 2010).