augmented lagrangian methods

For nonconvex optimization problems with nonlinear equality constraints, this paper proposes an inexact augmented Lagrangian method employing a norm penalty with exponent strictly between 1 and 2. The method constructs Hölder-smooth subproblems under convex feasibility and weak regularity assumptions, leveraging the first use of a non-integer-power Euclidean norm as the augmentation term. We establish, for the first time, accelerated first-order algorithm complexity bounds for such subproblems. Theoretically, we reveal an intrinsic trade-off: constraint violation converges faster as the exponent decreases, while dual residual decay remains controllably degraded. Numerical experiments demonstrate that the proposed method achieves superior constraint satisfaction accuracy and iteration efficiency compared to the standard squared-augmented Lagrangian method.

This paper addresses constrained nonconvex optimization problems involving both equality and inequality constraints. We propose an Adaptive Proximal Augmented Lagrangian Method (AP-ALM), which introduces a novel joint adaptive update rule for the penalty parameter and proximal term: rapid growth in early iterations to accelerate convergence, followed by controlled damping in later stages to mitigate ill-conditioning. Coupled with an inexact subproblem solver, the method ensures global convergence under mild assumptions. Theoretically, AP-ALM inherits the convergence guarantees of the classical Augmented Lagrangian Method (ALM) while relaxing the requirement for exact subproblem solutions. Numerical experiments demonstrate its robustness and efficiency on both convex and nonconvex benchmarks, achieving significantly faster convergence rates and enhanced numerical stability compared to state-of-the-art alternatives.

This work addresses the limitations of traditional numerical methods—which rely heavily on gradients and initial guesses—and the slow convergence of evolutionary algorithms in high-dimensional constrained optimization. To overcome these challenges, the paper proposes embedding a population-based stochastic optimizer, such as CMA-ES, into an augmented Lagrangian (AL) framework, replacing local solvers in AL subproblems with gradient-free global search. This approach represents the first systematic integration of the AL method’s robust constraint-handling capabilities with the strong exploratory power of evolutionary algorithms, effectively balancing feasibility enforcement and global exploration. Experimental results demonstrate that the proposed method significantly outperforms both pure evolutionary algorithms and state-of-the-art solvers like IPOPT on standard benchmark problems, particularly excelling in high-dimensional nonconvex landscapes riddled with numerous local minima and saddle points.

This paper addresses first-order optimization under nonlinear constraints—including nonconvex feasible sets—by proposing a novel accelerated algorithm grounded in nonsmooth dynamical systems. Methodologically, it models constraints in the **velocity space**, rather than the conventional position space, yielding sparse, local, and convex approximations of the feasible set and eliminating the need for expensive global projections at each iteration. Theoretically, the algorithm converges to stable points under nonconvex objectives and nonconvex constraints; under convexity, it achieves optimal acceleration rates in both continuous- and discrete-time settings. Its computational complexity scales nearly linearly with problem dimension and constraint count. Empirically, the method efficiently solves ℓ^p (p < 1) nonconvex regularized problems in compressed sensing and sparse regression: at p = 1, it matches state-of-the-art performance and substantially outperforms existing approaches.

This paper studies optimization problems with weakly convex objective functions subject to general convex (linear or nonlinear) constraints. We propose the Damped Proximal Augmented Lagrangian Method (DPALM), the first augmented Lagrangian framework incorporating a damping mechanism—specifically, damped dual step sizes—to ensure boundedness of dual variables and naturally accommodate nonlinear convex constraints. Theoretically, under weak convexity, DPALM achieves a tight outer-loop complexity of $O(varepsilon^{-2})$ and an overall complexity of $widetilde{O}(varepsilon^{-2.5})$ to attain a KKT residual $leq varepsilon$, improving upon the best-known bounds. Further integrating Moreau envelope smoothing with accelerated proximal gradient (APG) yields an overall complexity of $widetilde{O}(varepsilon^{-3})$. Experiments demonstrate that DPALM significantly outperforms state-of-the-art methods on nonconvex quadratic programming and robust nonlinear least-squares tasks.

This work addresses a class of nonconvex constrained optimization problems where both the objective and inequality constraints are compositions of convex Lipschitz outer functions with smooth inner mappings. The authors propose a smoothed proximal linear augmented Lagrangian method, reformulating the original problem as a nonsmooth nonconvex-concave minimax problem by restricting dual variables to a compact set. A finite-step mechanism is designed to map stationary points of the truncated minimax problem to KKT points of the original problem. Under a local cone regularity condition, they show that the artificial dual truncation automatically deactivates near feasible points, thereby establishing explicit convergence rates for the KKT residual: a global rate of $O(K^{-1/3})$ under dual regularization, which improves to $O(K^{-1/2})$ when the outer functions are piecewise linear and a local dual error bound holds.

This study investigates the statistical properties of Lagrange multipliers in constrained maximum likelihood estimation and least squares problems, along with their implications for numerical optimization. Leveraging large-sample theory, it establishes that under correctly specified models, Lagrange multipliers converge in probability to zero as the sample size grows, a result extended to high-dimensional settings such as deep learning. Building on this asymptotic behavior, the work provides the first statistical justification for initializing Lagrange multipliers at zero and integrates this insight into constrained optimization algorithms, including augmented Lagrangian methods and sequential quadratic programming. Numerical experiments demonstrate that this initialization strategy substantially enhances algorithmic stability and convergence efficiency in applications such as constrained regression and dynamic discrete choice models.

This work addresses the challenges of poor scalability, limited parallelizability, and complex subproblems in nonsmooth nonconvex optimization with orthogonality constraints by proposing a retraction-free primal-dual linearized smoothed augmented Lagrangian method. The proposed algorithm introduces, for the first time, a retraction-free primal-dual framework to orthogonality-constrained optimization, eliminating nested loops and intricate subproblem solvers in favor of a single-loop iteration scheme. Leveraging the Kurdyka–Łojasiewicz property, the method is theoretically shown to converge to an $\varepsilon$-KKT point with an iteration complexity of $O(\varepsilon^{-3})$, without requiring Riemannian retractions. Numerical experiments demonstrate that the algorithm significantly outperforms existing approaches in both computational efficiency and scalability.

This work addresses the challenge of integrating augmented Lagrangian and optimistic dual methods for equality-constrained optimization by proposing an additive hybrid framework that unifies matrix-valued augmentation and optimistic correction as distinct decompositions of a common correction matrix. By adaptively selecting the optimal splitting and stepsize through local spectral weighting, the method yields, for the first time, a closed-form hybrid update rule that jointly balances primal curvature and dual memory scale within a finite number of steps. Theoretical analysis reveals the equivalence and design flexibility between the two mechanisms, while experiments demonstrate that the proposed approach significantly outperforms individual strategies on nonlinear equality-constrained problems, achieving performance close to grid-search optimality and matching state-of-the-art first-order primal-dual algorithms under moderate ill-conditioning.

Existing end-to-end approaches to solving constrained convex optimization problems often fail to strictly satisfy constraints and lack guarantees of optimality. This work proposes a trainable architecture based on unfolded ADMM that enforces hard constraints through an embedded constraint-satisfaction module and a differentiable equality-constraint correction layer, ensuring exact feasibility at every iteration. Furthermore, first-order optimality conditions are incorporated as soft constraints into the training objective to guide convergence toward high-quality solutions. The proposed method uniquely unifies strict constraint satisfaction with optimality-aware learning within an unfolded optimization framework. Empirical results across multiple constrained convex optimization tasks demonstrate substantial improvements over conventional black-box end-to-end models, achieving both high solution accuracy and strong constraint compliance.