Comment on Ding and Miratrix: “To Adjust or Not to Adjust?”

I commend the authors for undertaking a detailed analysis of the bias produced by indiscriminate adjustments for pretreatment covariates [1]. While I agree with the analysis, I take exception to the authors’ conclusion that “for linear systems, except in some extreme cases, adjusting for all the pretreatment covariates is in fact a reasonable choice.” My reading of the analysis leads to the conclusion that indiscriminate adjustment is likely to introduce appreciable bias in causal effect estimates.

Ding and Miratrix (DM) divide their analysis into two parts:

1. Exact M-Structure, and

2. Deviations from M-structures.

I will follow this division and argue that: (1) the bias introduced by an exact M-structure is likely to be of the same order of magnitude as the bias that one attempts to eliminate by adjustment and (2) deviations from the exact M-structure may increase or decrease that bias, with roughly equal probabilities.

Starting with the exact M-structure, we learn that the bias introduced by adjusting for M calculates to Badj=|abcd|/[1−(bc)2], where a,b,c, and d are the correlation coefficients corresponding to figure 2 in DM’s paper. The bias B0 that one is concerned about, and that will remain in place if no adjustment is performed is roughly |adρ|. Thus, the ratio of the bias introduced to the bias one wishes to eliminate is

If one is operating in a highly noisy environment where b,c, and ρ are in the range 0.2−0.3, then the bias introduced by adjustment is roughly 20.8–32.9% of the confounding bias one is trying to eliminate. In a more deterministic environment, say where a,b, and ρ are in the range of 0.6–0.7, the bias introduced by adjustment would be 93.7–137% of the confounding bias one is trying to eliminate. The ratio increases dramatically at higher correlation coefficients. These figures hardly support DM’s conclusion that “the magnitude of M-bias in linear structural equation models tends to be relatively small compared to confounding bias”; I would describe the two biases as comparable.

Let us examine now the deviation from exact M-bias shown in DM’s figure 2, where confounding bias and M-bias coexist simultaneously. DM analyzed in details the case where a,b, and ρ are all positive, and have not sufficiently stressed the fact that the opposite effect will take place for negative bcρ, i.e., adjustment will cause an increase of bias for all values of b,c, and ρ. The reason is obvious; the two biases cancel each other when bcρ is positive, and reinforce each other when bcρ is negative.

Does nature prefer positive over negative correlations? I doubt it. Taxes are negatively correlated with consumer spending, and prices are negatively correlated with quantities consumed. Every prevention measure is negatively correlated with its outcome – police with crimes, fire fighters with fires, and so on. Thus, assuming that negative bcρ is as likely as positive bcρ, it would be an over-generalization to conclude that “mild deviations from the M-structure tend to increase confounding bias more rapidly than M-bias.” A more accurate summary would state that “mild deviations from the M-structure tend to increase or decrease the bias produced by conditioning on a collider.” Therefore, researchers should learn to detect M-bias and other bias-producing patterns in their models and decide, on a case by case basis, what covariates need be adjusted for. The ubiquity of M-bias in social science applications is demonstrated in Elwert and Winship [2]. Overall, I find Ding and Miratrix’s paper illuminating, and supportive of the methodological strategy expressed in [3, 4]: Justification of any model-blind method must rest on understanding model-specific analysis.