Transformed primal–dual methods for nonlinear saddle point systems
-
Long Chen
und
Jingrong Wei
Abstract
A transformed primal–dual (TPD) flow is developed for a class of nonlinear smooth saddle point systemThe flow for the dual variable contains a Schur complement which is strongly convex. Exponential stability of the saddle point is obtained by showing the strong Lyapunov property. Several TPD iterations are derived by implicit Euler, explicit Euler, implicit–explicit, and Gauss–Seidel methods with accelerated overrelaxation of the TPD flow. Generalized to the symmetric TPD iterations, linear convergence rate is preserved for convex–concave saddle point systems under assumptions that the regularized functions are strongly convex. The effectiveness of augmented Lagrangian methods can be explained as a regularization of the non-strongly convexity and a preconditioning for the Schur complement. The algorithm and convergence analysis depends crucially on appropriate inner products of the spaces for the primal variable and dual variable. A clear convergence analysis with nonlinear inexact inner solvers is also developed.
Funding statement: The authors were supported by NSF DMS-1913080 and DMS-2012465.
Acknowledgement
We would like to thank Dr. Jianchao Bai, Dr. Ruchi Guo, and Dr. Solmaz Kia for valuable suggestions, especially the discussion on the augmented Lagrangian methods. We also thank Dr. Hao Luo for careful proof reading and discussion on the Gauss–Seidel method with accelerated overrelaxation.
References
[1] K. J. Arrow, L. Hurwicz, and H. Uzawa. Studies in Linear and Non-Linear Programming. Stanford University Press, Stanford, CA, 1958.Suche in Google Scholar
[2] C. Bacuta. A unified approach for Uzawa algorithms. SIAM Journal on Numerical Analysis, 44 (2006) No. 6, 2633–2649.10.1137/050630714Suche in Google Scholar
[3] C. Bacuta. Schur complements on Hilbert spaces and saddle point systems. Journal of Computational and Applied Mathematics, 225 (2009), No. 2, 581–593.10.1016/j.cam.2008.08.025Suche in Google Scholar
[4] R. E. Bank, B. D. Welfert, and H. Yserentant. A class of iterative methods for solving saddle point problems. Numerische Mathematik, 56 (1989), No. 7, 645–666.10.1007/BF01405194Suche in Google Scholar
[5] M. Benzi, G. H. Golub, and J. Liesen. Numerical solution of saddle point problems. Acta Numerica, 14 (2005), 1–137.10.1017/S0962492904000212Suche in Google Scholar
[6] M. Benzi and M. A. Olshanskii. An augmented Lagrangian-based approach to the Oseen problem. SIAM Journal on Scientific Computing, 28 (2006), No. 6, 2095–2113.10.1137/050646421Suche in Google Scholar
[7] D. P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods. Academic Press, 2014.Suche in Google Scholar
[8] W. M. Boon, T. Koch, M. Kuchta, and K.-A. Mardal. Robust monolithic solvers for the Stokes–Darcy problem with the Darcy equation in primal form. SIAM Journal on Scientific Computing, 44 (2022), No. 4, B1148–B1174.10.1137/21M1452974Suche in Google Scholar
[9] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine learning}, 3 (2011), No. 1, 1–122.10.1561/2200000016Suche in Google Scholar
[10] J. Bramble, J. Pasciak, and A. Vassilev. Uzawa type algorithms for nonsymmetric saddle point problems. Mathematics of Computation, 69 (2000), 667–689.10.1090/S0025-5718-99-01152-7Suche in Google Scholar
[11] J. H. Bramble, J. E. Pasciak, and A. T. Vassilev. Analysis of the inexact Uzawa algorithm for saddle point problems. SIAM Journal on Numerical Analysis, 34 (1997), No. 3, 1072–1092.10.1137/S0036142994273343Suche in Google Scholar
[12] F. Brezzi. On the existence, uniqueness and approximation of saddle point problems arising from Lagrange multipliers. RAIRO Numerical Analysis, 8 (1974), 129–151.10.1051/m2an/197408R201291Suche in Google Scholar
[13] G. Chen and M. Teboulle. Convergence analysis of a proximal-like minimization algorithm using Bregman functions. SIAM Journal on Optimization, 3 (1993), No. 3, 538–543.10.1137/0803026Suche in Google Scholar
[14] L. Chen, X. Hu, and S. Wise. Convergence analysis of the fast subspace descent method for convex optimization problems. Mathematics of Computation, 89(2020), No. 325, 2249–2282.10.1090/mcom/3526Suche in Google Scholar
[15] L. Chen and H. Luo. A unified convergence analysis of first order convex optimization methods via strong Lyapunov functions. arXiv preprint arXiv:2108.00132, 2021.Suche in Google Scholar
[16] L. C̃hen and J. W̃ei. Accelerated gradient and skew-symmetric splitting methods for a class of monotone operator equations. arXiv preprint arXiv:2303.09009, 2023.Suche in Google Scholar
[17] L. Chen and Y. Wu. Convergence analysis for a class of iterative methods for solving saddle point systems. arXiv preprint arXiv:1710.03409, 2017.10.1007/s10915-018-0697-7Suche in Google Scholar
[18] L. Chen, Y. Wu, L. Zhong, and J. Zhou. Multigrid preconditioners for mixed finite element methods of the vector Laplacian. Journal of Scientific Computing, 77 (2018), No. 1, 101–128.10.1016/j.cam.2014.06.019Suche in Google Scholar
[19] P. Chen, J. Huang, and H. Sheng. Some Uzawa methods for steady incompressible Navier–Stokes equations discretized by mixed element methods. Journal of Computational and Applied Mathematics, 273 (2015), 313–325.10.1016/j.cam.2016.07.010Suche in Google Scholar
[20] P. Chen, J. Huang, and H. Sheng. Solving steady incompressible Navier–Stokes equations by the Arrow–Hurwicz method. Journal of Computational and Applied Mathematics, 311 (2017), 100–114.10.1137/S0036142995295789Suche in Google Scholar
[21] X. Chen. Global and superlinear convergence of inexact Uzawa methods for saddle point problems with nondifferentiable mappings. SIAM Journal on Numerical Analysis, 35 (1998), No. 3, 1130–1148.10.1016/S0377-0427(98)00197-6Suche in Google Scholar
[22] X. Chen. On preconditioned Uzawa methods and SOR methods for saddle-point problems. Journal of Computational and Applied Mathematics, 100 (1998), No. 2, 207–224.10.1137/S0036142998349266Suche in Google Scholar
[23] X.-L. Cheng. On the nonlinear inexact Uzawa algorithm for saddle-point problems. SIAM Journal on Numerical Analysis, 37(2000), No. 6, 1930–1934.10.1137/15M1026924Suche in Google Scholar
[24] A. Cherukuri, B. Gharesifard, and J. Cortes. Saddle-point dynamics: conditions for asymptotic stability of saddle points. SIAM Journal on Control and Optimization, 55(2017), No. 1, 486–511.10.1007/s10915-015-0048-xSuche in Google Scholar
[25] W. Deng and W. Yin. On the global and linear convergence of the generalized alternating direction method of multipliers. Journal of Scientific Computing, 66 (2016), No. 3, 889–916.10.1137/0731085Suche in Google Scholar
[26] H. C. Elman and G. H. Golub. Inexact and preconditioned Uzawa algorithms for saddle point problems. SIAM Journal on Numerical Analysis, 31 (1994), No. 6, 1645–1661.10.1137/0731085Suche in Google Scholar
[27] N. Golowich, S. Pattathil, C. Daskalakis, and A. Ozdaglar. Last iterate is slower than averaged iterate in smooth convex–concave saddle point problems. In: Conference on Learning Theory, PMLR, 2020, pp. 1758–1784.10.1515/9781400841042Suche in Google Scholar
[28] W. M. Haddad and V. Chellaboina. Nonlinear Dynamical Systems and Control: A Lyapunov-Based Approach. Princeton University Press, 2008.10.1090/S0025-5718-1978-0483340-6Suche in Google Scholar
[29] A. Hadjidimos. Accelerated overrelaxation method. Mathematics of Computation, 32 (1978), No. 141, 149–157.10.1090/S0025-5718-1978-0483340-6Suche in Google Scholar
[30] B. He and X. Yuan. Balanced augmented Lagrangian method for convex programming. arXiv preprint arXiv:2108.08554, 2021.10.1007/BF00927673Suche in Google Scholar
[31] M. R. Hestenes. Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4 (1969), No. 5, 303–320.10.1007/s002110100386Suche in Google Scholar
[32] Q. Hu and J. Zou. Two new variants of nonlinear inexact Uzawa algorithms for saddle-point problems. Numer. Math., 93 (2002), No. 2, 333–359.10.1137/S1052623403428683Suche in Google Scholar
[33] Q. Hu and J. Zou. Nonlinear inexact Uzawa algorithms for linear and nonlinear saddle-point problems. SIAM Journal on Optimization, 16 (2006), No. 3, 798–825.10.1007/s10915-012-9592-9Suche in Google Scholar
[34] B. Huang, S. Ma, and D. Goldfarb. Accelerated linearized Bregman method. Journal of Scientific Computing, 54 (2013), No. 2, 428–453.10.1007/s10915-017-0466-zSuche in Google Scholar PubMed PubMed Central
[35] J. Huang, L. Chen, and H. Rui. Multigrid methods for a mixed finite element method of the Darcy–Forchheimer model. Journal of Scientific Computing, 74 (2018), No. 1, 396–411.10.1007/s10915-017-0466-zSuche in Google Scholar
[36] H. K. Khalil. Nonlinear Systems, 3rd ed., Prentice-Hall, Upper Saddle River, NJ, 2002.Suche in Google Scholar
[37] G. M. Korpelevich. The extragradient method for finding saddle points and other problems. Matecon, 12 (1976), 747–756.10.1137/16M1097572Suche in Google Scholar
[38] X. Li, D. Sun, and K.-C. Toh. A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization, 28 (2018), No. 1, 433–458.10.1137/16M1097572Suche in Google Scholar
[39] H. Luo. Accelerated primal–dual methods for linearly constrained convex optimization problems. arXiv preprint arXiv:2109.12604, 2021.10.1051/cocv/2022032Suche in Google Scholar
[40] H. Luo. A primal–dual flow for affine constrained convex optimization. ESAIM: Control, Optimisation and Calculus of Variations, 28 (2022), 33.10.1137/19M127375XSuche in Google Scholar
[41] A. Mokhtari, A. E. Ozdaglar, and S. Pattathil. Convergence rate of 𝓞(1/k) for optimistic gradient and extragradient methods in smooth convex–concave saddle point problems. SIAM Journal on Optimization, 30 (2020), No. 4, 3230–3251.10.1007/978-1-4419-8853-9Suche in Google Scholar
[42] Y. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course, Vol. 87. Springer Science & Business Media, 2003.10.1137/18M1208836Suche in Google Scholar
[43] Y. Notay. Convergence of some iterative methods for symmetric saddle point linear systems. SIAM Journal on Matrix Analysis and Applications, 40 (2019), No. 1, 122–146.10.1137/040606028Suche in Google Scholar
[44] J. Peters, V. Reichelt, and A. Reusken. Fast iterative solvers for discrete Stokes equations. SIAM Journal on Scientific Computing, 27 (2005), No. 2, 646–666.10.1007/BF01141092Suche in Google Scholar
[45] L. D. Popov. A modification of the Arrow–Hurwicz method for search of saddle points. Mathematical Notes of the Academy of Sciences of the USSR, 28 (1980), No. 5, 845–848.10.1007/BF01141092Suche in Google Scholar
[46] M. J. Powell. A method for nonlinear constraints in minimization problems. Optimization, 1969, 283–298.10.1007/BF01588967Suche in Google Scholar
[47] M. J. Powell. Algorithms for nonlinear constraints that use Lagrangian functions. Mathematical Programming, 14 (1978), No. 1, 224–248.10.1109/LCSYS.2018.2851375Suche in Google Scholar
[48] G. Qu and N. Li. On the exponential stability of primal–dual gradient dynamics. IEEE Control Systems Letters, 3 (2018), No. 1, 43–48.10.1137/0726057Suche in Google Scholar
[49] W. Queck. The convergence factor of preconditioned algorithms of the Arrow–Hurwicz type. SIAM Journal on Numerical Analysis, 26 (1989), No. 4, 1016–1030.10.1137/19M1245736Suche in Google Scholar
[50] Y. Song, X. Yuan, and H. Yue. An inexact Uzawa algorithmic framework for nonlinear saddle point problems with applications to elliptic optimal control problem. SIAM Journal on Numerical Analysis, 57 (2019), No. 6, 2656–2684.10.1137/19M1293855Suche in Google Scholar
[51] Q. Tran-Dinh and Y. Zhu. Non-stationary first-order primal–dual algorithms with faster convergence rates. SIAM Journal on Optimization, 30 (2020), No. 4, 2866–2896.10.1137/090760350Suche in Google Scholar
[52] W. Yin. Analysis and generalizations of the linearized Bregman method. SIAM Journal on Imaging Sciences, 3 (2010), No. 4, 856–877.10.1137/090760350Suche in Google Scholar
[53] T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex–concave minimax problems with 𝓞(1/k2) rate on squared gradient norm. In: International Conference on Machine Learning, PMLR, 2021, pp. 12098–12109.Suche in Google Scholar
[54] X. Zeng, J. Lei, and J. Chen. Dynamical primal–dual Nesterov accelerated method with applications to network optimization. IEEE Transactions on Automatic Control, 68 (2022), No. 3, 1760-1767.10.1007/s10107-021-01660-zSuche in Google Scholar
[55] J. Zhang, M. Hong, and S. Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming, 194 (2022), No. 1, 901–935.10.1090/S0025-5718-01-01324-2Suche in Google Scholar
[56] W. Zulehner. Analysis of iterative methods for saddle point problems: a unified approach. Mathematics of Computation, 71 (2002), No. 238, 479–505.10.1137/100814767Suche in Google Scholar
[57] W. Zulehner. Nonstandard norms and robust estimates for saddle point problems. SIAM Journal on Matrix Analysis and Applications, 32 (2011), No. 2, 536–560.10.1137/100814767Suche in Google Scholar
© 2023 Walter de Gruyter GmbH, Berlin/Boston
Artikel in diesem Heft
- Frontmatter
- Research Article
- A posteriori error estimates for hierarchical mixed-dimensional elliptic equations
- Transformed primal–dual methods for nonlinear saddle point systems
- Diagonally implicit Runge–Kutta schemes: Discrete energy-balance laws and compactness properties
- New non-augmented mixed finite element methods for the Navier–Stokes–Brinkman equations using Banach spaces
Artikel in diesem Heft
- Frontmatter
- Research Article
- A posteriori error estimates for hierarchical mixed-dimensional elliptic equations
- Transformed primal–dual methods for nonlinear saddle point systems
- Diagonally implicit Runge–Kutta schemes: Discrete energy-balance laws and compactness properties
- New non-augmented mixed finite element methods for the Navier–Stokes–Brinkman equations using Banach spaces
