The robust isolated calmness of spectral norm regularized convex matrix optimization problems

1 Introduction

Consider the spectral norm regularized matrix optimization problem

where h : ℜ d → ℜ is twice continuously differentiable convex function and is essentially strictly convex, Q : ℜ m × n → ℜ d and ℬ : ℜ m × n → ℜ l are two linear operators, C ∈ ℜ m × n and b ∈ ℜ l are given data, and P ⊆ ℜ l is a given convex polyhedra cone. The function ‖ X ‖ 2 is the spectral norm function, namely, the largest singular value of X . Without loss of generality, we suppose that m ≤ n in what follows. For convenience, in the following text, we always use θ to represent the spectral norm function, i.e., θ ( ⋅ ) = ‖ ⋅ ‖ 2 . Problem (1.1) has a wide range of applications in various fields, such as matrix approximation problems [1], matrix Chebyshev polynomial problems [2], and H ∞ synthesis problems [3–5], etc. Recently, spectral norm regularized problem has also been applied in deep learning and neural networks problems [6–8].

Stability analysis theory is very crucial in studying the convergence of algorithms. As one of the important concepts in stability properties, isolate calmness (Definition 1) may guarantee the linear convergence rate of some algorithms, such as the alternating direction method of multipliers [9] and the proximal augmented Lagrangian method [10]. There are many publications on studying the isolated calmness of the Karush-Kuhn-Tucke (KKT) mapping (ICKKTM) for optimization problems. For the nonlinear semidefinite programming, Zhang and Zhang [11] show that the ICKKTM can be derived from the second-order sufficient condition (SOSC) and the strict Robinson constraint qualification (SRCQ). Zhang et al. [12] demonstrate that both the SOSC and the SRCQ are sufficient and necessary for the ICKKTM for the nonlinear second-order cone programming problem. It is important to note that Ding et al. [13] study the robust isolated calmness of a large class of cone programming problems. They prove that if and only if both the SOSC and the SRCQ hold, the KKT mapping is robustly isolated calm.

When the optimization problem has some kind of special linear structure, by establishing the equivalence between the constraint qualification of the primal (dual) problem and second-order optimality condition of the dual (primal) problem, one can obtain more characterizations of stability properties. For instance, for the standard semidefinite programming problem, Chan and Sun [14] show that the constraint nondegeneracy for the primal (dual) problem is equivalent to the strong SOSC for the dual (primal), then they obtain a series of equivalent characterizations of the strong regularity of the KKT point. Han et al. [15] discover the equivalence relationship between the dual (primal) SRCQ and the primal (dual) SOSC for convex composite quadratic semidefinite programming problem, thereby deriving a series of equivalent conditions of ICKKTM. For the case where θ in problem (1.1) is the nuclear norm function, Cui and Sun [16] show that the primal (dual) SRCQ is equivalent to the dual (primal) SOSC. Therefore, they derive more equivalent conditions of the robust ICKKTM.

Inspired by the work in [16], given the widespread application of spectral norm regularized convex matrix optimization problems, a natural question is whether the results in [16] can be extended to spectral norm regularized convex optimization problems. We provide an positive answer in this article. Due to the special structure of the critical cone of spectral norm function, we provide several equivalent conditions of the robust ICKKTM. That is to say, the conclusions in [16] are still valid for problem (1.1). Compared to [16], the difference in this article is that we use the established variational properties of the proximal mapping of spectral norm to provide a direct proof of the ICKKTM for problem (1.1), while [16] provides an indirect proof of the isolated calmness based on [13, Theorem 24] regarding the optimization problems with a C 2 -cone reducible constraint set. In addition, compared to the SOSC, the SRCQ for primal or dual problem is easier to verify, which can enhance the practicality of isolated calmness in algorithm research.

The organization of the subsequent content is as follows. We provide some notations and preliminaries on variational analysis, which will be used in the following text in Section 2. The variational properties of the spectral norm function are studied in Section 3, including the relationship between the critical cones of the spectral norm function and its conjugate function, and the explicit expression of the directional derivative of the proximal mapping of the spectral norm function. The results in Section 3 play a vital role in obtaining the important conclusions of this article. In Section 4, we prove that the primal (dual) SRCQ holds if and only if the dual (primal) SOSC holds and thus establish more equivalent conditions for the robust ICKKTM for problem (1.1). We conclude this article in Section 5.

Some common symbols and notations for matrices are as follows:

• For any positive integer t , we denote by [ t ] the index set { 1 , … , t } . For any Z ∈ ℜ m × n , the ( i , j ) th entry of Z is denoted as Z i j , where i ∈ [ m ] , j ∈ [ n ] . Let μ ⊆ [ m ] and ν ⊆ [ n ] be two index sets. We write Z ν to be the submatrix of Z that only retains all columns in ν , and Z μ ν to be the ∣ μ ∣ × ∣ ν ∣ submatrix of Z that only retains all rows in μ and columns in ν .

• For any d ∈ ℜ m , Diag ( d ) represents the m × m diagonal matrix, where the i th diagonal element is d i , i ∈ [ m ] .

• Using “trace” to represent the sum of diagonal elements in a given square matrix. For any two matrices P and Q in ℜ m × n , the inner product of P and Q is written as ⟨ P , Q ⟩ ≔ trace ( P T Q ) . The Hadamard product of matrices P and Q is represented by the symbol “ ∘ ”, i.e., the ( i , j ) th entry of P ∘ Q ∈ ℜ m × n is P i j Q i j .

• Let S w be the linear space of all w × w real symmetric matrices, and S + w and S − w be the cones of all w × w positive and negative semidefinite matrices, respectively.

2 Notation and preliminaries

Let X and Y be two finite dimensional real Euclidean spaces. Let B Y be the unit ball in Y . Let D ⊆ X be a nonempty closed convex set. For any z ∈ D , the tangent cone [17, Definition 6.1] of D at z is defined by T D ( z ) ≔ { d ∈ X ∣ ∃ z k → z with z k ∈ D and τ k ↓ 0 s.t. ( z k − z ) ⁄ τ k → d } . We use N D ( x ) ≔ { d ∈ X : ⟨ d , z − x ⟩ ≤ 0 , ∀ z ∈ D } to denote the normal cone of D at x ∈ D . Denote by δ D ( x ) the indicator function over D , i.e., δ D ( x ) = 0 if x ∈ D , and δ D ( x ) = ∞ if x ∉ D . Define the support function of the set D as σ ( y , D ) ≔ sup x ∈ D ⟨ x , y ⟩ , y ∈ X . For a given x ∈ X , define Π D ( x ) ≔ arg min { ‖ d − x ‖ ∣ d ∈ D } as the projection mapping onto D . Suppose the function f : X → ( − ∞ , + ∞ ] is proper closed convex, denote by dom f ≔ { x ∣ f ( x ) < ∞ } the effective domain of f , by f * the conjugate of f and by ∂ f the subdifferential of f . For more details, one can refer standard convex analysis [17]. For any convex cone K ⊂ Y , denote by K ∘ ≔ { z ′ ∈ Y ∣ ⟨ z ′ , z ⟩ ≤ 0 , ∀ z ∈ K } the polar of K .

The definition of isolated calmness below, which is taken from [13, Definition 2], is the most important concept in this article.

For any closed convex function g : X → ( − ∞ , + ∞ ] , we know from [18, Proposition 2.58] that g is directionally epidifferentiable. We use g ↓ ( x ; ⋅ ) to denote the directional epiderivatives of g . Further, if g ↓ ( x ; d ) is finite for x ∈ dom g and d ∈ X , we define the lower second-order directional epiderivative of g for any w ∈ X by

3 Variational analysis of the spectral norm function

In this section, we will provide a direct expression for the directional derivative of the proximal mapping of the spectral norm function. Then discuss the relationship between directional derivatives of the proximal mapping, the so-called sigma term and the critical cones.

Given an arbitrary matrix Q ∈ ℜ m × n , let σ 1 ( Q ) ≥ σ 2 ( Q ) ≥ … ≥ σ m ( Q ) be the singular values of Q . Denote σ ( Q ) ≔ ( σ 1 ( Q ) , σ 2 ( Q ) , … , σ m ( Q ) ) T . For any integer p > 0 , let O p be the set of all p × p orthogonal matrices. We assume that Q ∈ ℜ m × n admits the singular value decomposition (SVD) as follows:

where U ∈ O m and [ V 1 V 2 ] ∈ O n with V 1 ∈ ℜ n × m and V 2 ∈ ℜ n × ( n − m ) . Define the following three index sets

For any two matrices P , W ∈ S n , the well known Fan’s inequality [19] takes the form

where λ ( P ) represents the eigenvalue vector of P whose elements are arranged in nonincreasing order.

For any P , W ∈ ℜ m × n , by [20, Theorem 31.5], we know that W ∈ ∂ θ ( P ) (or, equivalently, P ∈ ∂ θ * ( W ) ) holds if and only if

where Prox θ : ℜ m × n → ℜ m × n denote the proximal mapping of θ [21, Definition 6.1], namely,

and Prox θ * : ℜ m × n → ℜ m × n denote the proximal mapping of θ * . Denote Q ≔ P + W . Let Q have the SVD as (3.1). Then by [21, Theorem 7.29 and Example 7.31], we have that

and

where κ : ℜ m → ℜ m is the proximal mapping of the l ∞ norm (i.e., the maximum absolute value of the elements in a vector in ℜ m ).

Define ϕ : ℜ → ℜ as the following scalar function:

where λ * > 0 and satisfies ∑ i = 1 m [ σ i ( Q ) − λ * ] + = 1 (for any x ∈ ℜ , [ x ] + means the nonnegative part of x ). Then, we can conclude that κ ( σ ) = ( ϕ ( σ 1 ) , ϕ ( σ 2 ) , … , ϕ ( σ m ) ) and

Obviously, Prox θ ( ⋅ ) can be regarded as Löwner’s operator related to ϕ .

Let ν 1 ( Q ) , ν 2 ( Q ) , … , ν r ( Q ) be the nonzero different singular values of Q , and there exists an integer 1 ≤ r ˜ ≤ r such that

To proceed, we further define several index sets. We denote a r + 1 ≔ b for convenience. Divide the set a into subsets as follows:

and define two index sets associated with λ * :

Specifically, σ i ( P ) = min { σ i ( Q ) , λ * } , i = 1 , … , m , namely,

In fact, λ * coincides with the largest singular value of P , namely, λ * = ‖ P ‖ 2 . According to Watson [22], the subdifferential of θ can be characterized as follows:

Therefore, we know that σ i ( W ) = 0 when i ∈ β , and the set α can be divided into three subsets as follows:

Hence, there exist integers 0 ≤ r 0 ≤ 1 and r ˜ − 1 ≤ r 1 ≤ r ˜ such that

Since the spectral norm function is globally Lipschitz continuous with modulus 1 and convex, θ is directional differentiable at any point in ℜ m × n . From (3.5), for any D ∈ ℜ m × n , the directional derivative of θ at P in the direction D can be expressed as follows:

For W ∈ ∂ θ ( P ) , or, equivalently, P ∈ ∂ θ * ( W ) , the critical cone of θ at P for W and the critical cone of θ * at W for P are defined as follows:

and

Next we will give the expression for the directional derivative of Prox θ ( ⋅ ) . By [23], we know that Prox θ ( ⋅ ) is directional differentiable and the directional derivative can be obtained via the directional derivative of ϕ . Clearly the directional derivative of ϕ is

For any Q ∈ ℜ m × n , denote Θ α α 2 : ℜ m × n → ℜ ∣ α ∣ × ∣ α ∣ , Θ α β 1 : ℜ m × n → ℜ ∣ α ∣ × ∣ β ∣ , Θ α β 2 : ℜ m × n → ℜ ∣ α ∣ × ∣ β ∣ , and Θ α c : ℜ m × n → ℜ ∣ α ∣ × ∣ c ∣ as follows:

Define two linear matrix operators S : ℜ p × p → S p and T : ℜ p × p → ℜ p × p by

For all Q ∈ ℜ m × n and D ∈ ℜ m × n , let D = [ D 1 D 2 ] with D 1 ∈ ℜ m × m , and D 2 ∈ ℜ m × ( n − m ) . We define four matrix mappings Ξ 1 : ℜ m × n × ℜ m × n → ℜ ∣ α ∣ × ∣ α ∣ , Ξ 2 : ℜ m × n × ℜ m × n → ℜ ∣ α ∣ × ∣ β ∣ , Ξ 3 : ℜ m × n × ℜ m × n → ℜ ∣ β ∣ × ∣ α ∣ , and Ξ 4 : ℜ m × n × ℜ m × n → ℜ ∣ α ∣ × ∣ c ∣ by

Therefore, according to [23, Theorem 3], the directional derivative Prox θ ′ ( Q ; D ) can be written as follows:

where D ˜ ≔ U T D V and

Theorem 3.1 in [24] shows that the singular value function is second-order directionally differentiable in ℜ m × n . This means that the spectral norm function is second-order directional differentiable, and θ ″ ( X ; ⋅ ) = θ ⇊ ( X ; ⋅ ) . To analyze the SOSC of problem (1.1), which helps to characterize the ICKKTM, we need to compute the “sigma term” of the problem (1.1), namely, the conjugate function of the second-order directional derivative of θ . For any k = 1 , … , r , define Ω a k : ℜ m × n × ℜ m × n → ℜ a k × a k by

where Z † is the Moore-Penrose pseudo-inverse of Z , Σ ( P ) ≔ Diag ( σ ( P ) ) , I m is the m × m identity matrix and D ˜ = [ D ˜ 1 D ˜ 2 ] = [ U T D V 1 U T D V 2 ] . By [25], we know that ∀ D ∈ ℜ m × n , the conjugate of θ ″ ( P ; D , ⋅ ) is

where σ k ( W ) = ( σ i ( W ) ) i ∈ a k . Clearly, ν k ( P ) = λ * when k = 1 , … , r ˜ . Continuing to calculate formula (3.14), we can obtain the explicit expression for ψ ( P , D ) * ( W ) as follows:

We are now prepared to give the main conclusions of this section, that is, some properties of the critical cones and the conjugate function of second-order directional derivative ψ * .

Propositions 10 and 12 in [25] characterized the critical cones of θ and θ * defined in (3.8) and (3.9), respectively. We summarize the results as follows.