augmented lagrangian optimization

For nonconvex optimization problems with nonlinear equality constraints, this paper proposes an inexact augmented Lagrangian method employing a norm penalty with exponent strictly between 1 and 2. The method constructs Hölder-smooth subproblems under convex feasibility and weak regularity assumptions, leveraging the first use of a non-integer-power Euclidean norm as the augmentation term. We establish, for the first time, accelerated first-order algorithm complexity bounds for such subproblems. Theoretically, we reveal an intrinsic trade-off: constraint violation converges faster as the exponent decreases, while dual residual decay remains controllably degraded. Numerical experiments demonstrate that the proposed method achieves superior constraint satisfaction accuracy and iteration efficiency compared to the standard squared-augmented Lagrangian method.

This paper addresses constrained nonconvex optimization problems involving both equality and inequality constraints. We propose an Adaptive Proximal Augmented Lagrangian Method (AP-ALM), which introduces a novel joint adaptive update rule for the penalty parameter and proximal term: rapid growth in early iterations to accelerate convergence, followed by controlled damping in later stages to mitigate ill-conditioning. Coupled with an inexact subproblem solver, the method ensures global convergence under mild assumptions. Theoretically, AP-ALM inherits the convergence guarantees of the classical Augmented Lagrangian Method (ALM) while relaxing the requirement for exact subproblem solutions. Numerical experiments demonstrate its robustness and efficiency on both convex and nonconvex benchmarks, achieving significantly faster convergence rates and enhanced numerical stability compared to state-of-the-art alternatives.

This work addresses the limitations of traditional numerical methods—which rely heavily on gradients and initial guesses—and the slow convergence of evolutionary algorithms in high-dimensional constrained optimization. To overcome these challenges, the paper proposes embedding a population-based stochastic optimizer, such as CMA-ES, into an augmented Lagrangian (AL) framework, replacing local solvers in AL subproblems with gradient-free global search. This approach represents the first systematic integration of the AL method’s robust constraint-handling capabilities with the strong exploratory power of evolutionary algorithms, effectively balancing feasibility enforcement and global exploration. Experimental results demonstrate that the proposed method significantly outperforms both pure evolutionary algorithms and state-of-the-art solvers like IPOPT on standard benchmark problems, particularly excelling in high-dimensional nonconvex landscapes riddled with numerous local minima and saddle points.

This paper studies optimization problems with weakly convex objective functions subject to general convex (linear or nonlinear) constraints. We propose the Damped Proximal Augmented Lagrangian Method (DPALM), the first augmented Lagrangian framework incorporating a damping mechanism—specifically, damped dual step sizes—to ensure boundedness of dual variables and naturally accommodate nonlinear convex constraints. Theoretically, under weak convexity, DPALM achieves a tight outer-loop complexity of $O(varepsilon^{-2})$ and an overall complexity of $widetilde{O}(varepsilon^{-2.5})$ to attain a KKT residual $leq varepsilon$, improving upon the best-known bounds. Further integrating Moreau envelope smoothing with accelerated proximal gradient (APG) yields an overall complexity of $widetilde{O}(varepsilon^{-3})$. Experiments demonstrate that DPALM significantly outperforms state-of-the-art methods on nonconvex quadratic programming and robust nonlinear least-squares tasks.

This paper studies variational inequality (VI) problems with multiple functional constraints. Conventional first-order methods rely on expensive projection or linear minimization oracles, while existing primal-dual algorithms require prior knowledge of optimal Lagrange multipliers. To address these limitations, we propose a purely primal Constraint Gradient Method (CGM), the first algorithm achieving non-asymptotic convergence without any prior information about Lagrange multipliers. Theoretically, under monotonicity and strong monotonicity assumptions, CGM attains the optimal operator query complexity—matching that of projection-based methods—while solving only lightweight quadratic programming subproblems per iteration, thereby substantially reducing computational overhead. Numerical experiments demonstrate that CGM is both efficient and robust on multi-constrained VI problems, consistently outperforming state-of-the-art primal-dual methods in practice.

This study investigates the statistical properties of Lagrange multipliers in constrained maximum likelihood estimation and least squares problems, along with their implications for numerical optimization. Leveraging large-sample theory, it establishes that under correctly specified models, Lagrange multipliers converge in probability to zero as the sample size grows, a result extended to high-dimensional settings such as deep learning. Building on this asymptotic behavior, the work provides the first statistical justification for initializing Lagrange multipliers at zero and integrates this insight into constrained optimization algorithms, including augmented Lagrangian methods and sequential quadratic programming. Numerical experiments demonstrate that this initialization strategy substantially enhances algorithmic stability and convergence efficiency in applications such as constrained regression and dynamic discrete choice models.

This work addresses a class of nonconvex constrained optimization problems where both the objective and inequality constraints are compositions of convex Lipschitz outer functions with smooth inner mappings. The authors propose a smoothed proximal linear augmented Lagrangian method, reformulating the original problem as a nonsmooth nonconvex-concave minimax problem by restricting dual variables to a compact set. A finite-step mechanism is designed to map stationary points of the truncated minimax problem to KKT points of the original problem. Under a local cone regularity condition, they show that the artificial dual truncation automatically deactivates near feasible points, thereby establishing explicit convergence rates for the KKT residual: a global rate of $O(K^{-1/3})$ under dual regularization, which improves to $O(K^{-1/2})$ when the outer functions are piecewise linear and a local dual error bound holds.

This work addresses the challenge of integrating augmented Lagrangian and optimistic dual methods for equality-constrained optimization by proposing an additive hybrid framework that unifies matrix-valued augmentation and optimistic correction as distinct decompositions of a common correction matrix. By adaptively selecting the optimal splitting and stepsize through local spectral weighting, the method yields, for the first time, a closed-form hybrid update rule that jointly balances primal curvature and dual memory scale within a finite number of steps. Theoretical analysis reveals the equivalence and design flexibility between the two mechanisms, while experiments demonstrate that the proposed approach significantly outperforms individual strategies on nonlinear equality-constrained problems, achieving performance close to grid-search optimality and matching state-of-the-art first-order primal-dual algorithms under moderate ill-conditioning.

This work addresses nonconvex equality-constrained optimization by proposing a gradient-eigenstep algorithm based on the Fletcher augmented Lagrangian function to efficiently compute approximate second-order stationary points. Under suitable initialization and parameter conditions, the algorithm is shown for the first time to enjoy local linear convergence in a neighborhood of strong second-order stationary points. Furthermore, when embedded as a subproblem solver within an incremental sampling strategy, the method significantly outperforms approaches that directly solve the full-sample problem for large-scale stochastic constrained optimization, thereby substantially reducing worst-case sample complexity.