convergence analysis

Linearly convergent optimization algorithms often exhibit suboptimal average-case performance on practical problem instances, yet their worst-case convergence guarantees—critical for safety-critical applications—must be preserved. Method: We propose a safety-enhancing framework grounded in nonsmooth composite optimization, which, for the first time, fully characterizes the class of admissible update-rule transformations that preserve linear convergence. This unified characterization encompasses foundational algorithms—including gradient descent, Nesterov’s accelerated method, and projection-based methods—as special cases. Contribution/Results: Our framework strictly maintains theoretical linear convergence rates while significantly accelerating empirical convergence in demanding settings such as ill-conditioned linear systems solving and model predictive control. Experiments demonstrate substantial improvements in average solution efficiency under finite iteration budgets, achieving both rigorous worst-case guarantees and strong practical performance.

This paper investigates the convexity conditions of optimization curves—i.e., the sequence of objective values {f(xₙ)}—generated by constant-step-size gradient descent on convex, L-smooth functions. The central question is: when is this sequence convex in the discrete sense, i.e., when are its second forward differences nonnegative—or equivalently, when is the forward difference f(xₙ) − f(xₙ₊₁) nonincreasing? The authors prove that strict convexity of the optimization curve holds if and only if the step size satisfies η ≤ 1.75/L, and this bound is tight (a counterexample shows convexity fails for any η > 1.75/L). Moreover, under the milder condition η ≤ 2/L, the gradient norm ∥∇f(xₙ)∥ is guaranteed to be nonincreasing. Leveraging forward-difference analysis, worst-case construction, and analogy with continuous-time gradient flow, the work establishes the first exact step-size characterizations for convexity and monotonicity in discrete optimization dynamics—thereby bridging a fundamental gap between discrete and continuous dynamical perspectives on convexity preservation.

There exists a significant gap between the theoretical convergence guarantees of deep learning optimization algorithms and their empirical performance, largely due to commonly adopted assumptions—such as Hessian boundedness—that lack empirical validation. Method: We introduce the first trajectory-aware measurement framework tightly aligned with key theoretical quantities, systematically evaluating the validity of mainstream assumptions across diverse architectures and datasets using large-scale training runs. Our framework quantifies dynamic properties—including gradient norms, Hessian spectral characteristics, and loss curvature—along optimization trajectories. Contribution/Results: We find that all examined theoretical assumptions fail to reliably predict actual convergence behavior and exhibit no robust correlation with optimization performance. This work uncovers a fundamental misalignment between theoretical modeling and practice, establishing the first reproducible benchmark for empirically calibrating and reconstructing optimization theory.

Existing iterative solvers for large-scale linear systems suffer from strong dependence on the global condition number and coarse-grained complexity analyses. Method: We introduce the *spectral tail condition number* $kappa_ell$, a new fine-grained spectral measure, and develop a refined time-complexity framework. Our approach formally defines $kappa_ell$, integrates it with the Sketch-and-Project paradigm, Nesterov acceleration, determinant point process sampling, and universality theory for Gaussian matrices, thereby exposing an intrinsic connection between iteration complexity and the matrix multiplication exponent $omega$. Contribution/Results: Our analysis achieves a sharper separation between deterministic and randomized algorithms, yielding an $ ilde{O}(kappa_ell n^2 log(1/varepsilon))$ bound for computing an $varepsilon$-accurate solution—valid for $ell$ up to $O(n^{0.729})$. This significantly improves the fine-grained analysis of the conjugate gradient method and establishes a novel theoretical benchmark for iterative algorithm design.

This work addresses the convergence guarantees of stochastic line search optimization for over-parameterized models under interpolation conditions. We establish a necessary and sufficient condition on the search direction—applicable to a broad class of methods—that ensures finite termination and bounded backtracking steps, and rigorously prove linear convergence under the Polyak–Łojasiewicz (PL) assumption. The condition unifies major first-order strategies—including momentum, conjugate gradient, and adaptive preconditioning—providing a verifiable theoretical foundation for their principled integration with stochastic line search. Our analysis fills a critical gap in the convergence theory of stochastic line search methods and significantly extends both the applicability and reliability of efficient first-order optimization in interpolation learning regimes.

This work addresses the lack of machine-verifiable formalizations of line search methods in nonlinear optimization, which has hindered algorithmic reliability. Within the Lean 4 theorem prover, it presents the first systematic formalization of several classical line search criteria—including Armijo, Goldstein, Wolfe, and their nonmonotone variants—alongside rigorous definitions of gradient descent, descent directions, and backtracking step-size selection. The study fully verifies the Zoutendijk convergence theorem within this framework, thereby establishing a comprehensive formal foundation for line search theory. This contribution significantly enhances the verifiability and trustworthiness of nonlinear optimization algorithms through mechanized mathematical reasoning.

This work proposes a trajectory-restricted framework for linear convergence analysis that overcomes the conservatism of traditional first-order methods, whose guarantees often rely on global geometric conditions and worst-case constants. Instead of imposing regularity assumptions globally, our approach requires only local geometric properties—such as restricted Polyak–Łojasiewicz inequalities, error bounds, and quadratic growth—on the subset of the space actually traversed by the algorithm. We establish explicit relationships among the associated constants and show that, for piecewise polyhedral composite problems, once iterates enter a well-conditioned active manifold, convergence is governed by the restricted Hoffman constant of that manifold, yielding an improved effective condition number and faster local convergence. The results demonstrate that linear convergence fundamentally depends on the local geometry encountered along the algorithmic trajectory, rather than on global worst-case scenarios.

This work addresses the finite-time convergence of stochastic iterative algorithms for fixed-point equations accessible only through a noisy oracle. The authors propose a norm-independent, unified Lyapunov function framework constructed via a generalized Moreau envelope, which integrates Lyapunov stability theory with stochastic approximation analysis. This framework accommodates complex settings such as Markovian noise, seminorm contractive operators, and dissipative operators, yielding sharp non-asymptotic convergence bounds in both high-probability and mean-square senses. As a result, it provides a unified and refined finite-time convergence guarantee for a broad class of algorithms, including stochastic gradient descent, linear stochastic approximation, Q-learning, and temporal difference learning.

This work addresses the limitations of classical iterative methods, which rely on forward error and are constrained by the condition number of the matrix. It introduces a new paradigm using backward error as the convergence criterion. The key contributions include the first proof that Richardson iteration achieves a condition-number-independent $O(1/k)$ convergence rate in backward error for any positive semidefinite linear system. Building on this, the authors design an accelerated algorithm, MINBERR, attaining an $O(1/k^2)$ convergence rate. They further integrate backward error minimization into Krylov subspace methods and extend the approach to general linear systems. The resulting general-purpose solver has complexity $O(n^2/\varepsilon)$, while MINBERR achieves $O(n^2/\sqrt{\varepsilon})$, demonstrating superior numerical performance in benchmark experiments.