Doubly weighted M-estimation for nonrandom assignment and missing outcomes

1 Introduction

When interest lies in causal inference, the prevalence of missing data poses a major identification challenge. In observational studies, causal effects estimation is complicated due to a nonrandom selection of individuals into programs or interventions (nonrandom treatment assignment). If, in addition, the observed outcome of interest is missing due to attrition or nonresponse, then this creates a double identification challenge for the estimation of treatment effects. This article proposes a class of doubly weighted M-estimators that are consistent and asymptotically normal for the joint problems of nonrandom treatment assignment and missing outcomes.

Despite the ubiquity of missing data problems in observational and experimental studies, the traditional inverse probability weighted (IPW) literature has only considered treatment and sample selection problems in isolation. In this article, I extend the literature to incorporate both issues simultaneously in a general framework. The main parameter of interest is defined to solve a population objective function. To correct for the two selection issues at hand, the article proposes weighting by both the propensity score and the missing outcome probability to identify the true parameter. The two key assumptions are unconfoundedness and missing at random, which represent ignorable assignment and missing data mechanisms, respectively [1]. Estimation follows in two steps: first the probabilities are estimated using binary response maximum likelihood, and second, the estimated probabilities are plugged in as weights to solve some objective function.

In the missing data IPW literature, Ding and Li [2] and Cao et al. [3] considered estimation of the population mean in the presence of missing outcomes. Robins and Rotnitzky [4] considered the estimation of regression parameters when the outcome is censored. Chen et al. [5] and Graham et al. [6] considered estimation of parameters indexing moment conditions, whereas Wooldridge [7] focused on the parameter solving an optimization problem. See the study by Seaman and White [8] for a review of IPW in the missing data context. While most of this literature draws parallels between missing data and missing potential outcomes, none have considered the problem of treatment selection. On the other hand, the causal inference literature uses propensity score weighting for identifying treatment effects but does not address any traditional missing data issues [9–13].

A few articles have looked at the double selection problem. For instance, Huber [14] presented a systematic treatment of the forms of attrition (i.e., whether selection is on observables or unobservables) that yield true average treatment effects (ATEs) in an experiment. His results under the “selection-on-observables” assumption are nested within this article as a special case. In another article, Huber [15] used an instrument for sample selection along with a conditionally exogeneous treatment. He then proposes a weighted estimator that nests the sample selection probability as an additional covariate in the propensity score. This is different from the weighting scheme employed in this article which neither involves any instruments nor nests probabilities. Rather, it exploits the sequential relationship between the two selection mechanisms; individuals may be more or less likely to attrit after selecting into a particular treatment group beyond what is dictated by their covariates. Other articles that take a selection-on-unobservables view of one or both problems include [16,17]. Typically, the discussion in these articles is limited to identification results for the ATE along with an incomplete characterization of the estimator in question. One exception is Huber [15] who also considered the unconditional quantile treatment effect (QTE) for the selected subpopulation.

This article makes the following contributions. First, it attempts to provide a comprehensive treatment of the two problems under a selection-on-observables framework where the parameter of interest minimizes an objective function. Consequently, the double-weighting solution applies to a wide range of procedures that can be framed as an M-estimation problem. The asymptotic theory of the proposed estimator is characterized in two parts. The first part describes cases where a conditional feature of interest is potentially misspecified, which is formalized in terms of a weak identification assumption. This part requires the weights to be correctly specified in order to achieve identification. In contrast, the second part assumes a conditional feature of interest to be correctly specified and allows the weights to be wrong. Together, they summarize two important cases of misspecification that can arise in this double-weighting setup and help us determine what can be consistently estimated under each setting. For instance, with respect to the estimation of QTEs, this article shows how one can estimate conditional QTE (CQTE) or a linear approximation to CQTE depending on whether the conditional quantile function is assumed to be correctly specified or not.

This article also shows that certain quasi-log-likelihood and mean function combinations deliver a “doubly robust” (DR) estimator of ATE under the given framework. This is different from the augmented-IPW style of estimators that are well known for being DR in the missing data literature ([18–20], see [21] for a review). DR estimators involve models for the conditional mean and propensity score and are consistent if at least one of two models is correctly specified. The extant literature on DR estimators have either focused on sample selection [2,6] or treatment selection [7,13], but not both at the same time [22]. A related contribution is to propose a DR estimator for ATE, which is distinct from the augmented-IPW class, when both problems are present. An advantage of this estimator is that it is less sensitive to extreme values of the weights and also ensures that the range of the estimated mean function aligns with the nature of the outcomes being studied [13]. Simulations show that the doubly weighted ATE and QTE estimates have the lowest finite sample bias compared to alternatives that ignore one or both problems. More recently, Bia et al. [23] have adapted the double machine learning framework of Chernozhukov et al. [24] to accommodate both treatment and sample selection problems in the presence of high-dimensional covariates.

This article also adds to the existing IPW literature on “efficiency puzzle” which finds that estimated nuisance parameters often provide a more efficient estimator for the main parameter of interest. This result appears while characterizing the variance of the proposed estimator under the first half of the asymptotic theory. The estimation of the weights helps to exploit the correlation between the first- and second-step moment conditions obtained from the binary response and M-estimation problems, respectively, which the known-weights estimator fails to do. However, this ceases to be true in the second half where there are no efficiency improvements from using estimated weights. This puzzle has been well studied in [7,25–27], and more recently in the studies by Lok [28] and Su et al. [29]. This article also discusses conditions when weighting may be inefficient compared to not weighting at all.

Finally, the proposed method is applied to estimate the average and distributional impacts of the National Supported Work (NSW) training program on earnings for the Aid to Families with Dependent Children (AFDC) target group. The sample was obtained from Calónico and Smith [30] who recreated Lalonde’s within-study analysis for this women’s sample. The presence of experimental and non-experimental comparison groups in the data help to evaluate whether the doubly weighted estimator brings us close to the experimental benchmark relative to other alternatives. I find that the empirical bias for the doubly weighted estimate is much smaller than that for the unweighted estimate.

The rest of this article proceeds as follows: Section 2 describes the basic potential outcomes framework and provides a short description of the population models with an introduction to the naive unweighted estimator; Section 3 discusses the treatment assignment and missing outcome mechanisms, which leads us directly to the identification lemma; Section 4 develops the first half of the asymptotic theory for the doubly weighted estimator with a focus on misspecification of a conditional feature of interest and correct weights; in contrast, Section 5 considers the second half where a conditional model of interest is correctly specified but the weights may be misspecified; Section 6 studies the estimation of average and QTEs within the proposed framework; Section 7 provides supporting Monte Carlo evidence under two cases of misspecification: the correct conditional model with misspecified weights and misspecified conditional model with correct weights; Section 8 applies the proposed method to the NSW job training program; and Section 9 concludes.

2 Potential outcomes and the population models

Let Y ( 1 ) and Y ( 0 ) denote potential outcomes for the treatment and control states, respectively, and let W be an indicator for the binary treatment. Then,

Also, let X be a fixed vector of covariates, which includes an intercept. Some feature of the distribution of ( Y ( g ) , X ) is assumed to depend on a finite vector θ g . Let q ( Y ( g ) , X , θ g ) be an objective function where some examples include the smooth least squares function, q ( ⋅ ) = ( Y ( g ) − X θ g ) 2 , or the non-smooth quantile regression, q ( ⋅ ) = c τ ( Y ( g ) − X θ g ) , where c τ ( u ) = ( τ − 1 { u < 0 } ) u is the asymmetric loss function for a random variable, u .

Assumption 1 defines the parameter of interest as the one that uniquely solves the population optimization problem. If q ( ⋅ ) involves a misspecified conditional feature like a conditional mean, variance, or even the full conditional distribution, Assumption 1 guarantees a unique pseudo-true solution [31]. In that case, determining whether the pseudo-truth, θ g 0 , is a meaningful parameter will depend on the conditional feature being studied and the estimation method used. For example, least squares can provide us with the best linear approximation to the true conditional mean, even if one mis-specifies the true function. Similarly, Angrist et al. [32] established the approximation properties of quantile regression where one can still estimate the best linear approximation to the true conditional quantile function under misspecification. In both cases, θ g 0 indexes linear projections (LPs) to different conditional models. On the other hand, when q ( ⋅ ) involves a correctly specified model, θ g 0 has a straightforward interpretation.

3 Identification of the parameter of interest

For identification of the main parameter, I make the following assumption.

This indicates that conditioning on covariates is enough to parse out any systematic differences between the treatment and control groups, also known as unconfoundedness. Part (i) requires that we observe these covariates for all individuals. Part (ii) is an overlap condition that ensures that we observe units in both the treatment and control groups for each value of x in the population. Previous literature has found several situations where unconfoundedness is a tenable assumption. This is especially true when pre-treatment values of the outcome variable are available. For example, Lalonde [33] and Hotz et al. [34] have shown that controlling for pre-training earnings alone reduces significant bias between non-experimental and experimental estimates. The literature assessing teacher impact on student achievement has reported similar findings with pre-test scores [35–37].

Assumption 3 is known as missing at random or MAR and represents an ignorable missing data mechanism. It implies that missingness depends only on observables and not on the missing values of the variable itself [38]. This includes missing completely at random as a special case [1]. The need to condition on W in addition to X helps to deal with the possibility that treatment itself can alter the probability of observing the outcome. This is especially useful in explaining cases of differential nonresponse. Parts (i) and (ii) have similar interpretations as before. Finally, Assumption 4 is a standard random sampling assumption.

Define ω 1 = S ⋅ W r ( X , W ) ⋅ p ( X ) , and ω 0 = S ⋅ ( 1 − W ) r ( X , W ) ⋅ ( 1 − p ( X ) ) for notational simplicity. Lemma 1 implies that solving the population problem with weights, ω g , is equivalent to solving the original M-estimation problem given in Assumption 1. The proof uses two applications of the law of iterated expectations (LIEs) with unconfoundedness and MAR to arrive at the above result.

4 Asymptotic theory when the conditional feature of interest is misspecified

Given that r ( X , W ) and p ( X ) are unknown, the following assumptions posit that we have a correctly specified model for the propensity score and missing outcome probability. Since both W and S are binary responses, estimation of γ 0 and δ 0 using maximum likelihood (MLE) will be asymptotically efficient under correct specification of these functions. Consistency and asymptotic normality for γ 0 and δ 0 follow from Theorems 2.5 and 3.3 of [39].

The “doubly weighted” estimator is then defined as follows:

where

are the estimated weights for solving the treatment and control group problems, respectively. Let d i and b i denote scores of the binary response log-likelihood problems for estimating the propensity score and missing outcome probability models evaluated at probability limits γ 0 and δ 0 , respectively. Also, let h ( θ g 0 ) ≡ h g denote the score of q ( ⋅ ) at the true parameter value and assume that it exists with probability one.

The asymptotic variance expression derived above offers some interesting insights. First, the middle term, Ω g , represents the variance of the residual from the population regression of the weighted score, l i g , on the two binary response scores, b i and d i . Note that the covariance term between the two MLE scores is zero since they are conditionally independent.

Second, the expression for Ω g has an efficiency implication for θ ˆ g . When one is only willing to assume identification of θ g 0 in the unconditional sense of Assumption 1, it is potentially more efficient to estimate the two weights even when they are known. To show this formally, let us assume that p ( X ) and r ( X , W ) are known and θ ˜ g is the doubly weighted estimator that uses known weights, ω g . Then,

In other words, we do no worse, asymptotically, by estimating the weights even when we actually know them. This result has also been called the “efficiency puzzle” and has been studied in Wooldridge [7] and others [26–29]. It is understood to arise from the suboptimal use of moment conditions in two-step procedures.

5 The conditional feature of interest is correctly specified

This section discusses the second half of the asymptotic theory for the doubly weighted estimator where identification of the parameter in question is formalized using a strong identification assumption.

Assumption 6 describes situations where a conditional feature of interest is correctly specified. This can be seen as strengthening the identification assumption in Section 4 since LIE implies that θ g 0 will also be a solution to the unconditional M-estimation problem.

An implication of this identification argument is that θ g 0 solves the conditional score of the objective function, i.e., E [ h g ∣ X ] = 0 . For instance, the conditional score will be zero when estimating a correctly specified conditional mean function using either least squares or quasi maximum likelihood in the linear exponential family. It would also hold for a correctly specified conditional quantile function estimated either using quantile regression or quasi maximum likelihood in the tick exponential family [40].

Under this conditional identification assumption, correct specification of the probability weights is not required for the doubly weighted estimator to be consistent for θ g 0 . In other words, R ( ⋅ , ⋅ , δ ) and G ( ⋅ , γ ) are allowed to be misspecified. Formally,

Under this setting, the weights are given by

where γ ˆ and δ ˆ solve the same binary response problems as before but converge to probability limits given by pseudo-true values γ ∗ and δ ∗ , respectively [31]. The identification argument in this case can be briefly explained as follows:

where ξ g ( X ) > 0 is a function of weights. If θ g 0 is a solution to the conditional problem E [ q ( ⋅ ) ∣ X ] , it will also solve equation (5) (multiplication by ξ g ( X ) will not affect the conditional minimization problem). Therefore, solving the doubly weighted objective function identifies the parameter even if the weights are misspecified. Theorem 2 establishes asymptotic results under this case.

Unlike the previous section, Ω g now is simply the variance of the weighted score of q ( ⋅ ) without the first-stage adjustment of the estimated probabilities. This is because under Assumption 6, the correlation between the score functions of the first- and second-step estimating equations is zero i.e., E ( l i g b i ′ ) = E ( l i g d i ′ ) = 0 . In other words, when θ g 0 is correctly specified for a conditional feature of interest and an appropriate estimation method is used, there is optimal usage of moment conditions.

A simpler expression for Ω g also means that we can no longer exploit the correlation between scores to obtain an efficient estimator of θ g 0 . Again, let θ ˜ g uses true weights, ω g . Then, we have the following result.

A special case is when ω g ∗ is just a constant since R ( ⋅ , ⋅ ) and G ( ⋅ , ⋅ ) are allowed to be any positive functions of X and W . This implies the unweighted estimator, θ ˆ g u , which does not weight at all, is also consistent for θ g 0 under the results of Theorem A.2. In this case, one may turn to asymptotic efficiency to guide our choice between weighting or not weighting at all. The following result says that if the objective function satisfies the generalized conditional information matrix equality (GCIME), the unweighted estimator is asymptotically more efficient than any weighted counterpart (correct weights or not).

The proof of this theorem follows from noting that we can express the difference in the two asymptotic variances as the expected outer product of population residuals from the regression of B i on D i , which are weighted versions of square root of matrix A i (see Appendix B for details). Hence, the difference is positive semi-definite.

We know GCIME in a variety of estimation contexts. In the case of full maximum likelihood, GCIME holds for q ( Y ( g ) , X , θ g ) = − ln f g ( Y ∣ X , θ g ) , where f g ( ⋅ ∣ ⋅ ) is the true conditional density with σ 0 g 2 = 1 . For estimating conditional mean parameters using quasi maximum likelihood estimation in the linear exponential family, GCIME holds if Var ( Y ( g ) ∣ X ) = σ 0 g 2 ⋅ v [ m ( X , θ g 0 ) ] . In other words, GCIME will be satisfied if Var ( Y ( g ) ∣ X ) satisfies the generalized linear model assumption, irrespective of whether the higher-order moments of the conditional distribution correspond with the chosen quasi-log likelihood or not. For estimation using nonlinear least squares, GCIME will hold for q ( Y ( g ) , X , θ g ) = [ Y ( g ) − m ( X , θ g ) ] 2 with the homoskedasticity assumption. In all these cases, the unweighted estimator will be more efficient than any weighted counterpart. Otherwise, the two may not be easy to rank.

6 Estimation of treatment effects

I now use asymptotic results discussed in Sections 4 and 5 for estimation of ATE and QTEs, which can be expressed as functions of the doubly weighted estimator, θ ˆ g .

7 Simulations

This section compares the empirical distributions of average and QTE estimators using unweighted, propensity score weighted (ps-weighted), and doubly weighted ( d -weighted) estimators. For estimating ATE, data are simulated using a probit as, Y ( g ) = 1 ( X θ g 0 + U ( g ) > 0 ) , where X includes an intercept and two covariates. For estimating the QTE parameters, I use an exponential data generating process where the potential outcomes are generated as Y ( g ) = exp [ X θ g 0 + U ( g ) ] , for g = 0 , 1 . For each setting, the covariates and latent errors have been drawn from two independent bivariate normal distributions. The treatment assignment and missing outcome mechanisms satisfy the assumptions of unconfoundedness and MAR with a 41% probability of being treated and 38% of being observed in the population, respectively. The empirical distributions of the unweighted, ps-weighted, and d -weighted estimators are then obtained for a sample of size 5,000 using 1,000 replication draws from a population of 1 million observations. Additional details can be found in Section S.1 of the Supplementary Material.

The discussion of the results is centered around two main misspecification scenarios: (1) when some conditional model (conditional mean function or conditional quantile function) is misspecified and (ii) when the weights are misspecified (enumerated in Tables S.1.1 and S.1.2). These correspond to scenarios outlined in the first and second half of the asymptotic theory.

7.1 ATE: Results

Case 1 in Figure 1 considers a misspecified mean function but correct probability weights. This is the principal case covered in Section 4 where weighting is crucial. As one can see, the empirical distribution of the doubly weighted estimator is centered on the true ATE, whereas the distribution for the unweighted estimator is shifted to the right.

Figure 1

Empirical distribution of estimated ATE for N = 5,000 . Notes: This figure plots the empirical distributions of the unweighted, ps-weighted, and d -weighted ATE estimates using 1,000 Monte Carlo simulation draws of sample size 5,000. The average treated sample size is N 1 = 5,000 × 0.41 × 0.38 = 779 , and the average control sample size is N 0 = 5,000 × ( 1 − 0.41 ) × 0.38 = 1 , 121 . The true ATE = 0.096, and the population is generated using a million observations. The unweighted estimator does not weight the observed data. The ps-weighted estimator weights to correct only for nonrandom assignment, and the d -weighted estimator weights by both the treatment and missing outcomes probabilities.

Case (2) of the same figure depicts the scenario of a correctly specified conditional mean function but misspecified weights. Here, weighting does not matter for consistent estimation of ATE since the mean function is correctly specified. All three empirical distributions, namely, unweighted, ps-weighted, and d -weighted, coincide and are centered on the true ATE.

7.2 QTE: Results

Figure 2 discusses the case when conditional quantile function is misspecified but the weights are correct. Using results obtained in the study by Angrist et al. [32], I interpret the solution to the double-weighted problem given in equation (12) as providing a consistent weighted linear approximation to the true conditional quantile function, which is then used to estimate an LP to the true CQTE. Figure 2 plots the bias in estimated LP using the three estimators relative to the true LP as a function of X 1 . We see that the double-weighted estimator performs the best.

Figure 2

Relative bias of the estimated LP (CQTE) as a function of X 1 when the conditional quantile function is misspecified but weights are correct. Notes: This figure plots the bias in the unweighted, ps-weighted, and d -weighted LPs to CQTE relative to the true population LP for N = 5,000 . The average treated sample size is N 1 = 5,000 × 0.41 × 0.38 = 779 , and the average control sample size is N 0 = 5,000 × ( 1 − 0.41 ) × 0.38 = 1,121 . The unweighted estimator does not weight the observed data. The ps-weighted estimator weights to correct only for nonrandom assignment, and the d -weighted estimator weights by the treatment and missing outcomes probabilities.

Next, I consider the case when the conditional quantile function is correctly specified but the weights are wrong. Since in this case one can consistently estimate the CQTE irrespective of the weights, Figure 3 plots the CQTE curve as a function of X 1 . In this case, as theory suggests, all three estimators of the CQTE function, i.e., unweighted, ps-weighted, and d -weighted, coincide with the true CQTE. Section S.1.2 of the Supplementary Material provides details about plotting the CQTE curve.

Figure 3

Estimated CQTE with true CQTE as a function of X 1 when conditional quantile function is correct but weights are misspecified. Notes: This figure plots the average d -weighted CQTE function with the true CQTE along X 1 for 1,000 Monte Carlo simulation draws of sample size N = 5,000 . Along with these two graphs, the figure also plots the individual function across the 1,000 simulation draws. The average treated sample is N 1 = 5,000 × 0.41 × 0.38 = 779 and average control sample is N 0 = 5,000 × ( 1 − 0.41 ) × 0.38 = 1,121 .

Finally, Figure 4 plots the empirical distribution of UQTE for the three estimators. We find that both unweighted and d -weighted estimators have a comparable finite sample bias. Propensity score weighting performs the worst in both cases. All results correspond to the twenty-fifth quantile of the outcome distribution. The results for other quantiles can be found in Section S.5 of the Supplementary Material.

Figure 4

Empirical distribution of estimated UQTE. Notes: This figure plots the empirical distributions of the unweighted, ps-weighted, and d -weighted UQTE estimates using 1,000 Monte Carlo simulation draws of sample size 5,000. The average treated sample is N 1 = 5,000 × 0.41 × 0.38 = 779 , and the average control sample is N 0 = 5,000 × ( 1 − 0.41 ) × 0.38 = 1,121 . The unweighted estimator does not weight the observed data. The ps-weighted estimator weights to correct only for nonrandom assignment, and the d -weighted estimator weights by both the treatment and missing outcomes propensity score models to deal with nonrandom assignment and missing outcome problems.

8 Returns to job training

In this section, I apply the proposed estimator to the AFDC sample of women from the NSW training program compiled by Calónico and Smith [30] (CS, thereafter). NSW was a transitional and subsidized work experience program, which was implemented as a randomized experiment in the United States between 1975–1979. CS replicate Lalonde’s [33] within-study analysis for the AFDC women in the program, where the purpose of such an analysis is to evaluate how training estimates obtained from using non-experimental identification strategies (assuming unconfoundedness) compare to experimental estimates. To compute the non-experimental estimates, CS combine the NSW experimental sample with two non-experimental comparison groups drawn from panel study of income dynamics (PSID), called PSID-1 and PSID-2. I utilize this within-study feature to estimate how close the d -weighted estimates are to the experimental benchmark compared with ps-weighted and unweighted estimates.

To construct these empirical bias measures, I first augment the CS sample to allow for women who had missing earnings information in 1979. This renders 26% of the experimental and 11% of the PSID samples missing. I then combine the experimental treatment group of NSW with three distinct comparison groups present in the CS dataset, namely, the experimental control group, and the two PSID samples, to compute the unweighted, ps-weighted, and d -weighted training estimates, respectively. The difference between the non-experimental estimate, obtained from using the d -weighted estimator, and the experimental estimate provides the first measure of estimated bias associated with the proposed strategy. Combining the experimental control group with the non-experimental comparison group gives a second measure of estimated bias [45]. I report both these bias measures for the average returns to training estimates.

Given the growing importance of estimating distributional impacts of job training programs, I also estimate returns to training at every tenth quantile of the 1979 earnings distribution. The role of double weighting is strong for estimating marginal quantiles since it serves to remove biases arising from the two selection problems.

8.1 Results

First, to evaluate whether women with missing earnings in 1979 were significantly different than those who were observed, Table 1 reports the mean and standard deviation of the woman’s age, years of schooling, pre-training earnings, and other characteristics across the observed and missing samples. In terms of age, the women who were observed in the experimentally treated group of NSW and the PSID-1 sample were, on average, older than those who were missing. The observed women in PSID-1 were also more likely to be married. For the PSID-2 sample, women who were observed had, on average, more kids with higher pre-training earnings. All these differences are statistically significant indicating that covariates were statistically different among the missing and observed PSID women (see the non-experimental columns in Table 1). For the experimental group, we do not find the covariates to be systematically different between those who were observed vs. those who were missing (see the experimental columns in Table 1).

Table 1

Covariate means and p -values from the test of equality of two means for the observed and missing samples

Covariates	Experimental						Non-experimental
	Control			Treatment			PSID-1			PSID-2
	Missing	Observed	P ( ∣ T ∣ > ∣ t ∣ )	Missing	Observed	P ( ∣ T ∣ > ∣ t ∣ )	Missing	Observed	P ( ∣ T ∣ > ∣ t ∣ )	Missing	Observed	P ( ∣ T ∣ > ∣ t ∣ )
Age, years	33.36	33.74	0.51	32.15	33.77	0.01	34.00	37.07	0.01	33.32	34.54	0.62
	(7.30)	(7.15)		(7.39)	(7.40)		(10.50)	(10.57)		(10.81)	(9.34)
Years of education	10.29	10.26	0.85	10.29	10.31	0.89	11.44	11.30	0.60	11.05	10.49	0.18
	(1.93)	(2.03)		(2.05)	(1.88)		(2.17)	(2.77)		(1.73)	(2.13)
Proportion of high school dropouts	0.70	0.68	0.57	0.69	0.70	0.77	0.43	0.45	0.73	0.55	0.59	0.68
	(0.46)	(0.47)		(0.46)	(0.46)		(0.50)	(0.50)		(0.51)	(0.49)
Proportion married	0.05	0.04	0.61	0.03	0.02	0.75	0.00	0.02	0.00	0.00	0.01	0.16
	(0.21)	(0.19)		(0.16)	(0.15)		(0.00)	(0.14)		(0.00)	(0.10)
Proportion Black	0.81	0.82	0.81	0.83	0.84	0.87	0.74	0.65	0.10	0.91	0.86	0.50
	(0.39)	(0.39)		(0.38)	(0.37)		(0.44)	(0.48)		(0.29)	(0.35)
Proportion Hispanic	0.12	0.13	0.87	0.13	0.12	0.64	0.01	0.02	0.82	0.05	0.02	0.62
	(0.33)	(0.33)		(0.33)	(0.32)		(0.11)	(0.12)		(0.21)	(0.15)
Number of children in 1975	2.33	2.23	0.34	2.14	2.19	0.69	1.54	1.71	0.33	2.41	2.97	0.05
	(1.29)	(1.34)		(1.32)	(1.29)		(1.45)	(1.78)		(1.14)	(1.79)
Real earnings in 1975	621.54	879.28	0.12	610.77	861.65	0.11	6927.95	7510.92	0.50	896.56	2211.45	0.02
	(1,523.00)	(2,194.93)		(1,677.36)	(2,005.53)		(7,330.74)	(7,541.41)		(2,315.12)	(3,567.50)
Observations	795	795		796	796		729	729		204	204

Notes: Along with the covariate means and standard deviation (in parentheses), the table also reports p -values from the test of equality for two means between the observed and missing samples. Real earnings in 1975 are expressed in terms of 1982 dollars.

The presence of non-experimental control groups implies that assignment was nonrandom and therefore an issue in the sample. This is because the comparison groups were drawn from PSID after imposing only a partial version of the full NSW eligibility criteria. Table 2 provides descriptive statistics for the covariates by the treatment status. As can be expected, the treatment and control groups of NSW are not observably different. In contrast, the women in PSID-1 and PSID-2 groups are statistically different from the treatment group.

Table 2

Covariate means and p -values from the test of equality of two means, by treatment status

Covariates	Treatment	Control	P ( ∣ T ∣ > ∣ t ∣ )	PSID-1	P ( ∣ T ∣ > ∣ t ∣ )	PSID-2	P ( ∣ T ∣ > ∣ t ∣ )
Age, years	33.37	33.64	0.46	36.73	0.00	34.41	0.11
	(7.42)	(7.19)		(10.60)		(9.48)
Years of education	10.30	10.27	0.72	11.32	0.00	10.55	0.07
	(1.92)	(2.00)		(2.71)		(2.09)
Proportion of high school dropouts	0.70	0.69	0.73	0.45	0.00	0.59	0.00
	(0.46)	(0.46)		(0.50)		(0.49)
Proportion married	0.02	0.04	0.03	0.02	0.05	0.01	0.08
	(0.15)	(0.20)		(0.13)		(0.10)
Proportion Black	0.84	0.82	0.29	0.66	0.00	0.87	0.13
	(0.37)	(0.39)		(0.47)		(0.34)
Proportion Hispanic	0.12	0.13	0.59	0.02	0.00	0.02	0.00
	(0.32)	(0.33)		(0.12)		(0.16)
Number of children in 1975	2.17	2.26	0.21	1.70	0.00	2.91	0.00
	(1.30)	(1.32)		(1.75)		(1.73)
Real earnings in 1975	799.88	811.19	0.91	7446.15	0.00	2069.65	0.00
	(1931.92)	(2041.32)		(7515.59)		(3474.10)
Observations	796	795		729		204

Notes: Along with the covariate means and standard deviation (in parentheses), the table also reports p -values from the test of equality for two means. Column 4 tests for differences between the NSW treatment and control groups, and columns 6 and 8 report the same using PSID-1 and PSID-2 comparison groups, respectively. Real earnings in 1975 are expressed in terms of 1982 dollars.

Table 3 reports the d -weighted, ps-weighted, and unweighted average returns to training estimates using three different comparison groups: NSW control, PSID-1, and PSID-2. The unweighted (unadjusted and adjusted) experimental estimates given in row 1 are the same as the estimates reported by CS in Table 3 of their article. Overall, one can see that the double-weighted experimental estimates are more stable than the single-weighted or unweighted estimates across the different regression specifications, with a range between $824 and $828. Moreover, the d -weighted estimator is DR for the ATE, which makes it more reliable as opposed to the other two.

Table 3

Unweighted and weighted earnings comparisons and estimated training effects using NSW and PSID comparison groups

Comparison group	Unadjusted			Adjusted			Adjusted
	Unweighted	PS-weighted	D-weighted	Unweighted	PS-weighted	D-weighted	Unweighted	PS-weighted	D-weighted
				Post-training earnings estimates
NSW	821	848	824	845	852	828	864	850	826
N = 1,185	(307.22)	(304.04)	(304.61)	(303.60)	(302.94)	(303.53)	(303.47)	(302.96)	(303.58)
PSID-1	− 799	827	803	298	909	907	335	905	904
N = 1,016	(444.84)	(503.00)	(503.26)	(428.60)	(497.76)	(501.54)	(440.18)	(518.54)	(522.97)
PSID-2	− 31	569	566	492	1,040	996	698	1,082	1,049
N = 720	(713.88)	(1041.81)	(1027.12)	(664.46)	(961.74)	(953.80)	(784.28)	(1264.18)	(1217.46)
				Bias estimates using NSW control
PSID-1	-1,620	169	156	− 493	− 40	− 21	− 568	− 38	− 21
N = 1,001	(431.75)	(561.74)	(553.07)	(427.93)	(499.91)	(501.44)	(434.59)	(504.19)	(507.02)
PSID-2	− 853	− 228	− 212	− 109	207	200	− 378	− 17	− 24
N = 705	(707.87)	(1041.44)	(1025.87)	(663.80)	(962.85)	(954.61)	(759.75)	(1195.47)	(1156.39)
Adjusted covariates
Pre-training earnings (1975)				✓	✓	✓	✓	✓	✓
Age				✓	✓	✓	✓	✓	✓
Age 2				✓	✓	✓	✓	✓	✓
Education				✓	✓	✓	✓	✓	✓
High school dropout				✓	✓	✓	✓	✓	✓
Black				✓	✓	✓	✓	✓	✓
Hispanic				✓	✓	✓	✓	✓	✓
Marital status				✓	✓	✓	✓	✓	✓
Number of Children (1975)							✓	✓	✓

Notes: This table reports unadjusted and adjusted post-training earnings differences between the NSW treatment and three different comparison groups, namely, NSW control, PSID-1 and PSID-2. The first row reports experimental training estimates that combine the NSW treatment and control group, whereas the second and third rows report non-experimental estimates computed from using the PSID-1 and PSID-2 groups, respectively. Each of the non-experimental estimates should be compared to the experimental benchmark. The second panel of the table reports bias estimates computed from combining the NSW control with PSID-1 and PSID-2 comparison groups, respectively. These represent a second measure of bias which should be compared to zero. Bootstrapped standard errors are given in parentheses and have been constructed using 10,000 replications. All values are in 1982 dollars. The samples used for estimating the training and bias estimates have been trimmed to ensure common support in the distribution of weights for the treatment and comparison groups. For more detail, see Section S.3 of the Supplementary Material.

For computing the ps-weighted and d -weighted non-experimental estimates, I first trim the sample to ensure common support between the treatment and comparison groups. Appendix S.3 describes estimation of the two probability weights along with the sample trimming criteria. This reduces the sample size from 1,248 to 1,016 observations for the PSID-1 estimates and from 782 to 720 observations for the PSID-2 estimates. A pattern that is consistent across the two sets of non-experimental estimates is that weighting gets us much closer to the benchmark relative to not weighting at all. For instance, the unweighted simple difference in means estimate of training, which uses the PSID-1 comparison group, is − $ 799 , whereas the weighted estimates are $827 and $803. For the PSID-2 comparison group, the unweighted estimate that controls for all covariates is $335, whereas the weighted estimates are $905 and $904.

The second panel of Table 3 reports the bias in training estimates from combining the experimental control group with the PSID comparison groups. A similar pattern is seen here with weighted bias estimates being much closer to zero than the unweighted estimates. These results suggest that the argument for weighting is strong when using a non-experimental comparison group where nonrandom assignment and missing outcomes are significant problems. Note that the large standard errors for the non-experimental estimates can be attributed to the small sample sizes and to the large residual variance of earnings in the PSID-1 and PSID-2 populations.

Figure 5 plots the relative bias in UQTE estimates at every tenth quantile of the 1979 earnings distribution. Much like the average training estimates, we see that the weighted estimates consistently lie below the unweighted estimates for most quantiles, irrespective of whether we use the PSID-1 or PSID-2 nonexperimental group.

Figure 5

Relative estimated bias in UQTE estimates at different quantiles of the 1979 earnings distribution. (a) PSID-1 control group and (b) PSID-2 control group. Notes: This graph plots the bias in the unweighted, ps-weighted, and d -weighted UQTE estimates relative to the true experimental estimates across different quantiles of the 1979 earnings distribution. Panel (a) plots the relative bias estimates using the PSID-1 comparison group and panel (b) plots the same using the PSID-2 comparison group. The treatment and missing outcome propensity score models have been estimated as flexible logits, and the samples used for constructing these estimates have been trimmed to ensure common support across the two groups. The treatment propensity score has been estimated using the full experimental sample along with either PSID-1 or PSID-2 comparison group. The UQTE estimates for τ < 0.46 are omitted from the graph since these are zero.

9 Conclusion

In empirical research, the problems of nonrandom assignment and missing outcomes threaten identification of treatment effects. This article proposes a class of M-estimators that double weight by the propensity score and missing outcome probability to correct for the two problems within a selection-on-observables framework. The asymptotic theory of the proposed estimator is characterized in two halves where the first half allows misspecification in some conditional feature of interest and the second allows for misspecified weights. Together, the two parts completely characterize the kinds of misspecification scenarios permissible under the given framework.

As illustrative examples, the article utilizes results from the first and second half to discuss estimation of causal parameters like the average and QTEs. In the case of ATE, the proposed estimator is shown to be DR irrespective of whether the mean function or the weights are misspecified (not both). For the case of QTEs, one may either obtain the CQTE if the conditional quantile function is assumed correct or a linear approximation to it. This is demonstrated in the simulations where we find that the double-weighted ATE and QTE estimates have the lowest bias when compared to naive alternatives (unweighted and ps-weighted estimators). Finally, an application of the procedure to Calónico and Smith’s (2017) reconstructed NSW sample helps to quantify the degree of distortion created by the two problems on the returns to training estimates through a comparsion with the experimental benchmark.

Even though missing outcomes are a common concern in empirical analysis, it is equally common to encounter missing data on the covariates. A particularly important future extension can be to allow for missing data on both. In this case, using a generalized method of moments framework, which incorporates information on complete and incomplete cases, could provide efficiency gains over just using the observed data. A different possibility would be to relax the identifying restrictions to allow for selection on unobservables and possibly explore estimation of local ATE.