On the Condition Number Dependency in Bilevel Optimization

2025-11-27

AI Summary

This work studies the oracle complexity of first-order algorithms for finding ε-stationary points in nonconvex–strongly-convex bilevel optimization, focusing on the optimal dependence on the lower-level condition number κ. Methodologically, it establishes the first provable complexity separation between bilevel optimization and minimax problems, introduces the first tight lower bound Ω(κ²/ε²), and—under a stochastic oracle model—derives a stronger lower bound Ω(κ⁴/ε⁴), substantially improving prior results. It further proposes a novel algorithm achieving an upper bound of Õ(κ^{7/2}/ε²). Through smoothing analysis, higher-order smoothness assumptions, and stochastic modeling, the work systematically characterizes the dependence on κ across diverse settings. The results provide the tightest known characterization of condition-number dependence in bilevel optimization complexity theory to date, revealing its intrinsic computational hardness.

Technology Category

Reasoning under Uncertainty: Stochastic OptimizationSearch and Optimization: Non-convex OptimizationMachine Learning: Optimization

Application Category

Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsEconomics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystemsSecurity and Privacy: Large-scale security measurements

Abstract

Bilevel optimization minimizes an objective function, defined by an upper-level problem whose feasible region is the solution of a lower-level problem. We study the oracle complexity of finding an $ε$-stationary point with first-order methods when the upper-level problem is nonconvex and the lower-level problem is strongly convex. Recent works (Ji et al., ICML 2021; Arbel and Mairal, ICLR 2022; Chen el al., JMLR 2025) achieve a $ ilde{mathcal{O}}(κ^4 ε^{-2})$ upper bound that is near-optimal in $ε$. However, the optimal dependency on the condition number $κ$ is unknown. In this work, we establish a new $Ω(κ^2 ε^{-2})$ lower bound and $ ilde{mathcal{O}}(κ^{7/2} ε^{-2})$ upper bound for this problem, establishing the first provable gap between bilevel problems and minimax problems in this setup. Our lower bounds can be extended to various settings, including high-order smooth functions, stochastic oracles, and convex hyper-objectives: (1) For second-order and arbitrarily smooth problems, we show $Ω(κ_y^{13/4} ε^{-12/7})$ and $Ω(κ^{17/10} ε^{-8/5})$ lower bounds, respectively. (2) For convex-strongly-convex problems, we improve the previously best lower bound (Ji and Liang, JMLR 2022) from $Ω(κ/sqrtε)$ to $Ω(κ^{5/4} / sqrtε)$. (3) For smooth stochastic problems, we show an $Ω(κ^4 ε^{-4})$ lower bound.

Problem

• Establishes new lower and upper bounds for bilevel optimization complexity.
• Investigates condition number dependency in nonconvex-strongly convex bilevel problems.
• Extends lower bounds to high-order smooth, stochastic, and convex hyper-objectives.

Innovation

• Establishes new lower bound for bilevel optimization complexity
• Proves gap between bilevel and minimax problem complexities
• Extends bounds to high-order smooth and stochastic settings