Optimal Stochastic Bilevel Optimization with First-Order Oracles

2026-10-01

AI Summary

This study addresses the unresolved question of computational complexity optimality for nonconvex-strongly-convex bilevel optimization under stochastic first-order oracles. To this end, it proposes MRT-FD, a single-loop algorithm that simultaneously updates upper- and lower-level variables while auxiliary-tracking the response mapping. By leveraging O(p)-th order finite differences to approximate second-order derivative information, the method avoids explicit Hessian computation. The primary contribution is establishing the first complexity lower bound for this problem setting, thereby closing the gap between existing upper and lower bounds for a fixed smoothness parameter p and confirming theoretical optimality. Specifically, MRT-FD requires only O(ε^{-4-2/p}) stochastic gradient queries to reach an ε-stationary point, matching the proven lower bound and achieving the optimal convergence rate in this setting.

Abstract

We study nonconvex--strongly-convex bilevel optimization under a stochastic first-order oracle. We introduce MRT-FD, a single-loop first-order method that simultaneously tracks the upper-level variable, the lower-level solution, and the auxiliary response arising from implicit differentiation of the hyperobjective. MRT-FD performs one update of each variable per iteration and approximates the second-order derivative actions using order-$p$ finite differences. For any fixed finite smoothness order $p\ge1$ in the lower-level variable, MRT-FD finds an $\varepsilon$-stationary point using $\mathcal{O}(\varepsilon^{-4-2/p})$ stochastic gradient queries. We also prove a matching $Ω(\varepsilon^{-4-2/p})$ oracle lower bound. The lower-bound construction starts from a hard nonconvex minimization chain with a stronger stochastic oracle, and lifts it to a bilevel problem through a sinusoidal coupling with a scalar lower-level variable. Consequently, the dependence on $\varepsilon$ is optimal for every fixed finite $p$, closing the upper--lower complexity gap in this stochastic first-order oracle setting.

Problem

• Bilevel Optimization
• Stochastic First-Order Oracle
• Nonconvex-Strongly-Convex
• Sample Complexity
• Oracle Lower Bound

Innovation

• Stochastic Bilevel Optimization
• Single-loop First-order Method
• Finite Differences
• Oracle Lower Bound
• Complexity Gap