optimize via successive convex approximation

This paper addresses first-order optimization under nonlinear constraints—including nonconvex feasible sets—by proposing a novel accelerated algorithm grounded in nonsmooth dynamical systems. Methodologically, it models constraints in the **velocity space**, rather than the conventional position space, yielding sparse, local, and convex approximations of the feasible set and eliminating the need for expensive global projections at each iteration. Theoretically, the algorithm converges to stable points under nonconvex objectives and nonconvex constraints; under convexity, it achieves optimal acceleration rates in both continuous- and discrete-time settings. Its computational complexity scales nearly linearly with problem dimension and constraint count. Empirically, the method efficiently solves ℓ^p (p < 1) nonconvex regularized problems in compressed sensing and sparse regression: at p = 1, it matches state-of-the-art performance and substantially outperforms existing approaches.

This work systematically uncovers the decisive role of problem geometry—specifically, the curvature of the constraint set and the structure of gradients—in governing the statistical-computational trade-offs of stochastic and online optimization algorithms. We introduce the first geometric measure quantifying the deviation of a constraint set from quadratic convexity, rigorously identifying the geometric origins of suboptimality in subgradient methods. We prove that diagonal-preconditioned SGD achieves minimax-optimal convergence rates under quadratic convex constraints. For non-Euclidean, non-quadratically-convex domains—such as ℓₚ-balls with p < 2—we establish tight convergence bounds for mirror descent and adaptive gradient methods, and uncover, for the first time, a precise correspondence between their convergence rates and the accuracy-computation trade-off in Gaussian sequence estimation. Our results provide geometric criteria for algorithm selection and unify the understanding of when nonlinear updates—e.g., via mirror descent—are necessary to attain statistical optimality.

For nonconvex optimization problems with nonlinear equality constraints, this paper proposes an inexact augmented Lagrangian method employing a norm penalty with exponent strictly between 1 and 2. The method constructs Hölder-smooth subproblems under convex feasibility and weak regularity assumptions, leveraging the first use of a non-integer-power Euclidean norm as the augmentation term. We establish, for the first time, accelerated first-order algorithm complexity bounds for such subproblems. Theoretically, we reveal an intrinsic trade-off: constraint violation converges faster as the exponent decreases, while dual residual decay remains controllably degraded. Numerical experiments demonstrate that the proposed method achieves superior constraint satisfaction accuracy and iteration efficiency compared to the standard squared-augmented Lagrangian method.

This work addresses the challenge of implicit constraints—manifested as evaluation failures—arising from unreliable physics-based simulations in system architecture optimization. To tackle this, we propose a surrogate modeling framework that integrates probabilistic feasibility prediction with Bayesian optimization. Methodologically, we introduce a novel hybrid discrete Gaussian process to model the Probability of Validity (PoV), coupled with an interior-point selection strategy based on a minimum PoV threshold; the framework natively supports hierarchical design variables and multi-objective optimization. Our approach achieves the first successful solution for a jet engine architecture optimization task with a 50% simulation failure rate. Across multiple synthetic benchmarks and real-world case studies, it significantly improves convergence robustness and optimization success rate. The implementation is publicly available as the SBArchOpt Python library.

This work investigates the dynamical behavior of Nesterov-type accelerated gradient methods—including variable- and constant-momentum NAG and NCM—in escaping strict saddle points and converging to local minima for smooth nonconvex optimization. Employing non-asymptotic and asymptotic dynamical systems analysis, Lyapunov function construction, and manifold characterization near saddle points, we establish the first rigorous proof that variable-parameter NAG almost surely avoids strict saddle points. We introduce two asymptotic rate metrics and derive linear-scale estimates for saddle-point escape time. Moreover, we identify a subclass of accelerated methods that simultaneously achieves near-optimal convergence rates and strong escape capability. Our results provide a unified characterization of escape performance and local convergence rates across mainstream acceleration algorithms, establishing a novel theoretical framework for designing and analyzing accelerated methods in nonconvex optimization.

This work addresses the challenge of efficiently solving parametric nonconvex optimization problems by proposing a learning-driven surrogate modeling approach. The method constructs a surrogate function expressed as the pointwise minimum of a finite set of convex and monotonic functions, thereby reformulating the original nonconvex problem into a collection of convex subproblems that can be solved in parallel. This approach achieves, for the first time, a structured convex approximation of nonconvex objectives, offering both high approximation accuracy and significantly improved computational efficiency. Numerical experiments on nonconvex path-following tasks demonstrate the superior performance of the proposed method in terms of both solution accuracy and computational speed.

This work addresses optimization problems with convex constraints whose intersection is difficult to project onto, covering both strongly convex smooth and general nonsmooth convex settings. The authors propose a novel algorithm that integrates stochastic feasibility methods with (sub)gradient descent, wherein each iteration randomly samples a subset of constraints and employs an adaptive Polyak stepsize that requires no prior knowledge of problem parameters, complemented by iterate averaging. Theoretical analysis establishes linear convergence under strong convexity and a worst-case rate of $O(1/\sqrt{T})$ for general convex objectives, while the infeasibility measure decays geometrically almost surely. Numerical experiments on QCQP and SVM tasks demonstrate superior computational efficiency over existing methods, and under specific sampling strategies, the algorithm achieves optimal convergence rates.

This work addresses the computational challenges of embedding neural networks into mathematical optimization, where conventional feedforward neural networks (FNNs) yield mixed-integer programming (MIP) reformulations that are computationally expensive and suffer from loose relaxations. To overcome these limitations, the paper proposes using input convex neural networks (ICNNs) as surrogate models, leveraging their inherent convexity to construct tight linear programming (LP) relaxations. The authors establish, for the first time, an exact convex hull-based continuous relaxation of ICNNs over box domains, yielding an LP representation free of integrality gaps. Furthermore, they introduce a novel branch-and-bound algorithm that branches directly on input variables. Demonstrated across applications in humanitarian food aid allocation, oil well trajectory planning, and wine blending, the approach achieves approximation accuracy comparable to FNNs while significantly improving solution speed and scalability.

This work addresses nonconvex equality-constrained optimization by proposing a gradient-eigenstep algorithm based on the Fletcher augmented Lagrangian function to efficiently compute approximate second-order stationary points. Under suitable initialization and parameter conditions, the algorithm is shown for the first time to enjoy local linear convergence in a neighborhood of strong second-order stationary points. Furthermore, when embedded as a subproblem solver within an incremental sampling strategy, the method significantly outperforms approaches that directly solve the full-sample problem for large-scale stochastic constrained optimization, thereby substantially reducing worst-case sample complexity.

This work addresses the challenges of poor scalability, limited parallelizability, and complex subproblems in nonsmooth nonconvex optimization with orthogonality constraints by proposing a retraction-free primal-dual linearized smoothed augmented Lagrangian method. The proposed algorithm introduces, for the first time, a retraction-free primal-dual framework to orthogonality-constrained optimization, eliminating nested loops and intricate subproblem solvers in favor of a single-loop iteration scheme. Leveraging the Kurdyka–Łojasiewicz property, the method is theoretically shown to converge to an $\varepsilon$-KKT point with an iteration complexity of $O(\varepsilon^{-3})$, without requiring Riemannian retractions. Numerical experiments demonstrate that the algorithm significantly outperforms existing approaches in both computational efficiency and scalability.