Reply to Professor Pearl’s Comment

We thank Professor Pearl for his interest in our paper on M-Bias, and would like to clarify a few points we were attempting to make.

First, we would like to clarify that in the case of an exact M structure, there is no bias to remove, and so adjusting will indeed induce bias. We argue that this bias is generally small in size, but of course there is no complete standard for measurement. In our paper we give a few special cases. For example, if the correlations for the M graph in Figure 2 are constrained as a=b and c=d, the bias induced by adjustment satisfies |Bias|<min(|a|,|c|)/20, i.e., is smaller than the one twentieth of the minimum value of |a| and |c| for the grey area of Figure 4(a). While Pearl’s observation that large values of (a,b,c,d) can result in large biases is indeed correct, the square defined by |a|<0.5 and |c|<0.5 is almost entirely grey: for systems that are not nearly deterministic, M-Bias is usually very small.

Pearl, in his comment, compares the bias of no adjustment in the case when the unmeasured variables U and V are correlated with correlation ρ to the bias from adjustment in the pure M-bias case (where ρ=0). We believe it is more comparable to examine no adjustment vs. adjustment for fixed ρ, which corresponds to the first formula in Section 2.2 of our paper. Here, if we let a,b,c,d, and ρ all equal 0.2, say, in magnitude, and arrange the signs of these to maximize bias from adjustment, we find that bias can be increased by about 17%, which is of course not ideal. However, as all the parameters grow, the bias ratio hovers near this 20%, rather than climbing markedly. In fact, as Figure 5 shows, larger ρ only increases the range of parameters for which adjustment is superior (in the positive bcρ case). Only if ρ is relatively small and the other correlations are large do you pay a large penalty for adjustment. However, even these 20% increases depend on the product bcρ being negative, which warrants further attention. We discuss this next.

Pearl doubts that “nature prefer[s] positive over negative correlations,” and we whole-heartedly agree. Negative correlations per-se are not uncommon. However, the product of bcρ being negative means the correlations have to hold a specific three-way relationship. To see the form of this relationship assume, without loss of generality, that ρ>0. Then negative bcρ requires U and W affect M in opposite directions even though they are positively associated with each other. We did not find such a relationship in our survey of examples in the literature of M-Bias. For example, consider Figure 11(a), the Rubin–Pearl controversy example. Attitudes toward societal norms, U, is positively associated with attitudes toward safety and health related measures, W. Both of them affects seat-belt usage, M, in the same direction. In this case, bcρ is positive. All three examples in Figure 11 have the same pattern with bcρ>0. Because of this, we did not, in Figure 5, present the results with positive bcρ. In hindsight, we should have, along with further discussion.

In sum, we agree with Professor Pearl that M-Bias is likely to be large if the system is close to deterministic or bcρ is negative. Given this, we agree that it is important for researchers investigating a problem of interest to assess whether the system is close to deterministic, and whether negative bcρ is possible. But, in the social sciences at any rate, deterministic systems are fairly rare, and bcρ being negative requires a fairly specific interaction with latent traits. Given this we generally feel that, in an uncertain world, adjustment is more likely to help than hurt. Nevertheless, we do agree that if possible some checks to assess whether material bias could be introduced by adjustment would be worthwhile.