Asymmetric Laplace Regression: Maximum Likelihood, Maximum Entropy and Quantile Regression

A. Interpretation of the Z-estimator

In order to interpret θ₀, we take the expectation of the estimating equations with respect to the unknown true density. To simplify the exposition we consider a simple model without covariates: y_i=α+u_i. Our estimating equation vector is defined as:

E(Ψθ(y))=E(1σ(τ−1(y<α))(1−2ττ(1−τ)−(y−α)σ)−1σ+1σ2ρτ(y−α))=0,

and the estimator is such that

1n∑i=1nψθ(yi)=0

Let F(y) be the cdf of the random variable y. Now we need to find E[Ψ_θ(y)].

For the first component we have

1σE[τ−I(y<α)]=1σ(∫ℝ(τ−1(y<α))dF(y))=1σ(τ−∫−∞αdF(y))=1σ(τ−F(α)).

Thus if we set this equal to zero, we have

α=F−1(τ),

which is the usual quantile. Thus, the interpretation of the parameter α is analogous to QR if covariates are included.

For the third term in the vector, −1σ+1σ2ρτ(y−α), we have

E[−1σ+1σ2ρτ(y−α)]=0,

that is,

σ=E[ρτ(y−α)].

Thus, as in the least squares case, the scale parameter σ can be interpreted as the expected value of the loss function.

Finally, we can interpret τ using the second equation,

E[1−2ττ(1−τ)−(y−α)σ]=0,

which implies that

1−2ττ(1−τ)=E[y]−F−1(τ)σ.

Note that s(τ)≡1−2ττ(1−τ) is a measure of the skewness of the distribution. Thus, τ should be chosen to set s(τ) equal to a measure of asymmetry of the underline distribution F(·) given by the difference of τ-quantile with the mean (and standardized by σ). In the special case of a symmetric distribution, the mean coincides with the median and mode, such that E[y]=F^–1(1/2) and τ=1/2, which is the most probable quantile and a solution to our Z-estimator.

B. Lemma A1

In this appendix we state an auxiliary result that states Donskerness and stochastic equicontinuity. Let ℱ≡{ψθ(y, x), θ∈Θ}, and define the following empirical process notation for w=(y, x):

f↦En[f(w)]=1n∑i=1nf(wi)  f↦Gn[f(w)]=1n∑i=1n(f(wi)−Ef(wi)).

We follow the literature using empirical process exploiting the monotonicity and boundedness of the indicator function, the boundedness of the moments of x and y, and that the problem is a parametric one.

Lemma A1 Under Assumptions A1–A4 ℱ is Donsker. Furthermore,

θ↦Gnψθ(y, x)

is stochastically equicontinuous, that is

sup||θ−θ0||≤δn||Gnψθ(y, x)−Gnψθ0(y, x)||=op(1),

for any δ_n↓0.

Proof: The proof of this result follows similar steps to those in Chernozhukov and Hansen (2006). To prove the lemma we check the conditions for independent but not identically distributed process stated in Theorem 2.11.1 of van der Vaart and Wellner (1996). It is important to note that a class ℱ of a vector-valued functions f:x↦ℝk is Donsker if each of the classes of coordinates fj:x↦ℝk with f=(f₁, …, f_k) ranging over ℱ(j=1, 2,...,k) is Donsker (van der Vaart 1998, 270).

First, one can check the random-entropy condition by checking that ℱ satisfies a uniform entropy condition and the envelope is square integrable. The first element of the vector is ψ1θ(y, x)=(τ−1(y<x′β))xσ. Note that the functional class 𝔄={τ−1{y<x′β}, τ∈T, β∈ℬ} is a VC subgraph class, with envelope 2. Its product with x also forms a class with a square integrable envelope 2 max_j|x_j|. Finally, the class ℱ1 is defined as the product of the latter with 1/σ, which is bounded by assumption A3. Thus, by assumption A4 ℱ1 is satisfies a uniform entropy condition and the envelope function is square integrable. Therefore, the random entropy condition (2.11.2) in van der Vaart and Wellner (1996) is satisfied.

The second element of the vector is ψ2θ(y, x)=(1−2ττ(1−τ)−(y−x′β)σ). Define 𝕳={(y−x′β), β∈ℬ}. Note that

|(y−x′β1)−(y−x′β2)|=|x′(β2−β1)|≤||x||||β2−β1||,

where the inequality follows from Cauchy-Schwartz inequality. Thus by Assumptions A3–A4 the class 𝕳 has envelope function square integrable. In addition, note that, 𝕳 belongs to a VC class satisfying a uniform entropy condition, since this class is a subset of the vector space of functions spanned by (y, x₁, …, x_p), where p is the fixed dimension of x (see e.g., example 19.7 in van der Vaart (1998)). Thus, the class defined by 1/σ𝕳  has envelope of 𝕳  (|y|+const*|x|) which is square integrable by assumptions A3–A4. Therefore, ℱ2 satisfies the random entropy condition.

The third element of the vector is ψ3θ(y, x)=(−1σ+1σ2ρτ(y−x′β)). Consider the following random process defined by 𝕵={ρτ(y−x′β), τ∈T, β∈ℬ} which satisfies a uniform entropy condition and square integrable envelope function. The latter is given by Assumptions A3–A4 and the quantile regression check function properties as ρ_τ(x+y)–ρ_τ(y)≤2|x| and ρτ1(y−x′t)−ρτ2(y−x′t)=(τ2−τ1)(y−x′t). The former follows from the fact that this is a parametric class collection of measurable functions indexed in a bounded subset. Hence, ℱ3 the random entropy condition and the envelope function is square integrable.

Now we turn our attention to the second condition of Theorem 2.11.1 in van der Vaart and Wellner (1996). The process θ↦Gnψθ(y, x) is stochastically equicontinuous over Θ with respect to a L₂(P) pseudo-metric. First, as in Angrist, Chernozhukov, and Fernández-Val (2006) and Chernozhukov and Hansen (2006), we define the distance d as the following L₂(P) pseudo-metric

d(θ′, θ′′)=E([ψθ′−ψθ′′]2).

Thus, as ||θ–θ₀||→0 we need to show that

and the final follows from Theorem 2.11.1 of van der Vaart and Wellner (1996).

To show (28), first note that for each i=1, …, n,

d1i(θ′, θ)=E([ψ1θ′−ψ1θ]2)=E([(τ′−1(yi−xiβ′))xiσ′−(τ−1(yi−xiβ))xiσ]2)≤[(E|1σ′(τ′−1(yi−xiβ′))−1σ1(τ−(yi−xiβ))|2(2+ϵ)ϵ)ϵ(2+ϵ)⋅(E(|xi|2)2+ϵ2)2(2+ϵ)]12=(E|(τ′σ′−τσ)+(1σ1(yi≤xiβ)−1σ′1(yi≤xiβ′))|2(2+ϵ)ϵ)ε2(2+ϵ)⋅(E(|xi|2)2+ϵ2)1(2+ϵ)≤[((|τ′σ′−τσ|)2(2+ϵ)ϵ)ϵ2(2+ϵ)+(E(|1σ1(yi≤xiβ)−1σ′1(yi≤xiβ′)|)2(2+ϵ)ϵ)ϵ2(2+ϵ)]⋅(E(|xi|2)2+ϵ2)1(2+ϵ)≤[|τ′σ′−τσ|+(E|g¯i⋅xi(β′σ′−βσ)|)ϵ2(2+ϵ)]⋅(E||xi||2+ϵ)1(2+ϵ)≤[|τ′σ′−τσ|+(E g¯i||xi||β′σ′−βσ||)ϵ2(2+ϵ)]⋅(E||xi||2+ϵ)1(2+ϵ),

where the first inequality is Holder’s inequality, the second is Minkowski’s inequality, the third is a Taylor expansion as in Angrist, Chernozhukov and Fernández-Val (2006), p.560) where g̅_i is the upper bound of g_i(y_i|x) (using A2), and the last is Cauchy-Schwarz inequality. Therefore, by assumption A2–A4 sup||θ′−θ||<δn∑i=1nd1i→0 when δ_n→0.

Now rewrite ψ2θ(y, x)=(σ1−2ττ(1−τ)−(y−x′β)) and note that

d2i(θ′,θ)=E([ψ2θ′−ψ2θ]2)=E([σ′1−2τ′τ′(1−τ′)−(yi−x′iβ′)−σ1−2ττ(1−τ)+(yi−x′iβ)]2)=E(|σ′1−2τ′τ′(1−τ′)−σ1−2ττ(1−τ)+(x′i(β−β′))|2)≤(|σ′1−2τ′τ′(1−τ′)−σ1−2ττ(1−τ)|2)1/2+(E|xi′(β−β′)|2)1/2≤(|σ′1−2τ′τ′(1−τ′)−σ1−2ττ(1−τ)|2)1/2+||β′−β||(E||xi||2)1/2,

where the first inequality is given by Minkowski’s inequality (E|X+Y|^p)^1/p≤(E|X|^p)^1/p+(E|Y|^p)^1/p for p≥1, and the second inequality is Cauchy-Schwarz inequality. Hence, assumptions A3–A4 ensure that sup||θ′−θ||<δn∑i=1nd2i→0 when δ_n→0.

Finally, rewrite ψ_3θ(y, x)=(–σ+ρ_τ(γ–x′β)), and thus

d3i(θ′, θ)=E([ψ3θ′−ψ3θ]2)=E([−σ′+ρτ′(yi−xiβ′)+σ−ρτ(yi−xiβ)]2)=E([−σ′+σ+ρτ′(yi−xiβ′)−1σ2ρτ(yi−xiβ)]2)≤(−σ′+σ)2+E([ρτ′(yi−xiβ′)−ρτ(yi−xiβ)]2)=(−σ′+σ)2+E([ρτ′(yi−xiβ′)−ρτ′(yi−xiβ)+ρτ′(yi−xiβ)−ρτ(yi−xiβ)]2)≤|σ−σ′|+E([||xi(β′−β)||+|τ′−τ|(yi−xiβ)]2)≤|σ−σ′|+E([||xi||||β′−β||+|τ′−τ|(yi−xiβ)]2)≤|σ−σ′|+(E[||xi||||β′−β||]2)1/2+(E[|τ′−τ|(yi−xiβ)]2)1/2=|σ−σ′|+||β′−β||(E||xi||2)1/2+|τ′−τ|(E[(yi−xiβ)]2)1/2≤const⋅(|σ−σ′|+||β′−β||+|τ′−τ|),

where the first inequality is given by Minkowski’s inequality, the second inequality is given again by QR check function properties as ρ_τ(x+y)–ρ_τ(y)≤2|x| and ρτ1(y−x′t)−ρτ2(y−x′t)=(τ2−τ1)(y−x′t). Third inequality is Cauchy-Schwarz inequality. Fourth is Minkowski’s inequality. Last inequality uses assumption A4, and finally we have that sup||θ′−θ||<δn∑i=1nd3i→0 when δ_n→0.

Thus, ||θ′–θ||→0 implies that d(θ′, θ)→0 in every case, and therefore, sup||θ′−θ||<δn∑i=1ndi→0 when δ_n→0. The final condition in Theorem 2.11.1 in van der Vaart and Wellner (1996) is a Lindeberg condition, which is guaranteed by assumptions A1–A4. Therefore, we conclude that ℱ is Donsker and

sup||θ−θ0||≤δn||Gnψθ(y, x)−Gnψθ0(y, x)||=op(1).■