lipschitz constant estimation

This work analyzes the linear convergence rate of the (1+1)-Evolution Strategy (ES) on locally $L$-strongly convex functions with $U$-Lipschitz continuous gradients. Unlike existing derivative-free optimization theory—which relies on prior knowledge of problem parameters (e.g., $L$, $U$)—this paper establishes the first tight exponential convergence bounds *without any parameter information*: an upper bound of $exp(-Omega(L/(d cdot U)))$ and a matching lower bound of $exp(-1/d)$. Methodologically, we integrate a stochastic adaptive step-size mechanism, probabilistic convergence analysis, and asymptotic reasoning as dimension $d o infty$, thereby explicitly characterizing how the convergence rate depends on both dimension $d$ and condition number $L/U$. This result breaks the reliance of classical black-box optimization theory on strong prior assumptions, providing the first parameter-free, dimensionally explicit theoretical guarantee for the (1+1)-ES.

This paper investigates the convexity conditions of optimization curves—i.e., the sequence of objective values {f(xₙ)}—generated by constant-step-size gradient descent on convex, L-smooth functions. The central question is: when is this sequence convex in the discrete sense, i.e., when are its second forward differences nonnegative—or equivalently, when is the forward difference f(xₙ) − f(xₙ₊₁) nonincreasing? The authors prove that strict convexity of the optimization curve holds if and only if the step size satisfies η ≤ 1.75/L, and this bound is tight (a counterexample shows convexity fails for any η > 1.75/L). Moreover, under the milder condition η ≤ 2/L, the gradient norm ∥∇f(xₙ)∥ is guaranteed to be nonincreasing. Leveraging forward-difference analysis, worst-case construction, and analogy with continuous-time gradient flow, the work establishes the first exact step-size characterizations for convexity and monotonicity in discrete optimization dynamics—thereby bridging a fundamental gap between discrete and continuous dynamical perspectives on convexity preservation.

This work addresses the challenge of verifying Lipschitz constants in conventional neural networks, which typically relies on computationally expensive methods or overly loose trivial bounds that fail to guarantee adversarial robustness and generalization. The authors propose a novel “verification-by-training” paradigm that integrates structural design to directly optimize and tighten trivial Lipschitz bounds during training, thereby circumventing complex post-hoc verification. Key innovations include norm-saturating polynomial activations (polyactivations), unbiased sinusoidal layers, and extensions to non-Euclidean norms, collectively eliminating three major sources of bound looseness. On MNIST, the resulting networks achieve Lipschitz bounds several orders of magnitude lower than existing approaches, with less than 10% error relative to the true Lipschitz constant, significantly enhancing both robustness and generalization performance.

Existing methods for estimating the Lipschitz constant of deep neural networks suffer from a trade-off between accuracy and scalability: semidefinite programming (SDP)-based approaches incur prohibitive computational cost and poor scalability, whereas closed-form methods yield overly conservative bounds. This work introduces a novel class of closed-form, scalable Lipschitz upper bounds. By generalizing the parameterization of the LipSDP feasible region and integrating matrix norm inequalities with layer-wise propagation bound optimization, our method enables cooperative utilization of multiple parameterized feasible points—without invoking an SDP solver—for the first time. The approach unifies and generalizes ECLipsE-Fast, achieving significantly tighter bounds and higher computational efficiency on large-scale networks. It supports real-time robustness verification for models with up to hundreds of millions of parameters.

This paper studies the oracle complexity of finding a (δ,ε)-stable point—a point whose δ-neighborhood contains a subgradient of norm at most ε—in nonsmooth optimization. Under Lipschitz continuity, it establishes for the first time that deterministic first-order algorithms necessarily incur dimension-dependent complexity, precluding dimension-free bounds; in contrast, randomized algorithms achieve the tight upper bound Õ(1/(δε³)), matched by a universal randomized lower bound. It further reveals that convexity dramatically accelerates convergence: for convex functions, a deterministic O(1/ε²) upper bound is attained, and it is proven that zero subgradients cannot be identified exactly in finite time; for smooth functions, derandomization is achievable with only logarithmic overhead. The core contribution lies in precisely characterizing the fundamental roles of determinism vs. randomness, convexity, and smoothness in governing oracle complexity, and providing tight upper and lower bounds for each setting.

This study investigates the computational complexity of computing the Lipschitz constant of the solution mapping for multiparametric quadratic programs—a quantity essential for optimization-based control analysis. By leveraging computational complexity theory, APX-hardness analysis, and parameterized complexity techniques, the work establishes for the first time that this problem is not only NP-hard but also APX-hard, even in the scalar parameter case. Nevertheless, the problem becomes polynomial-time solvable when either the number of constraints or the number of decision variables is fixed. These theoretical findings demonstrate that the intrinsic difficulty stems from the number of constraints and decision variables rather than the dimensionality of the parameters. Numerical experiments corroborate the validity of these conclusions.

This work proposes a dual-agent collaborative framework for automatically discovering convex relaxations to strengthen lower bounds in nonconvex optimization problems. An encoding agent generates tight constraints, while a theory agent validates their correctness through explicit dual feasible points and rigorous interval arithmetic. The approach pioneers the integration of large language model–driven autonomous research paradigms into convex relaxation construction, unifying automated lower-bound optimization with formal mathematical proof. The method achieves new state-of-the-art results on two classical optimization constants: improving $C_{6.2}$ from 1.28 to 1.2937 and $C_{6.5}$ from 0.379005 to 0.37912.

This work investigates the optimal query complexity lower bounds for first-order gradient oracles in finding ε-stationary points of higher-order smooth nonconvex optimization problems. By introducing a novel “block-chain” mechanism to construct hard instances and integrating higher-order smoothness analysis with deterministic first-order complexity theory, the study establishes the first dimension-free tight lower bounds: Ω(ε⁻⁷/⁴) under Hessian-Lipschitz continuity and Ω(ε⁻⁵/³) in the third-order smooth setting. These bounds match the known upper bounds achieved by existing accelerated algorithms, thereby resolving a long-standing gap in the theoretical understanding of nonconvex optimization complexity.

This work addresses a long-standing theoretical gap in deterministic derivative-free convex optimization by establishing the first near-tight lower bound on query complexity when only exact function evaluations are available. Specifically, for minimizing convex Lipschitz functions over the $d$-dimensional Euclidean ball, the paper proves a deterministic query complexity lower bound of $\Omega(d^2 / \log(d+1))$, closing the gap that has persisted for nearly three decades up to a logarithmic factor. The result is achieved through a carefully constructed adversarial family of functions, combined with tools from information theory and convex analysis. Furthermore, the authors extend their approach to the mixed-integer setting, yielding a complexity bound of $\widetilde{O}(d^2 \cdot 2^n)$, which significantly improves upon existing estimates.

This work addresses the limitations of classical algorithmic stability theory, which typically relies on strong assumptions such as bounded loss functions or sub-Gaussian/sub-Weibull tail behavior—conditions often violated in heavy-tailed or unbounded loss settings. The paper introduces a novel $L_p$ stability framework that requires only finite $L_p$ moments of the loss function, thereby relaxing the conventional bounded differences condition. By extending McDiarmid’s inequality to accommodate $L_p$ constraints, the authors derive sharp high-probability generalization bounds under this significantly weaker assumption. This approach is shown to be broadly applicable across multiple learning paradigms, including empirical risk minimization, transductive regression, and meta-learning, demonstrating that robust generalization guarantees can still be achieved even when losses are unbounded, provided $L_p$ stability holds.