Optimal Momentum Methods for Stochastic Multilevel Compositional Optimization

2026-10-01

AI Summary

This study addresses the challenge of stochastic multilevel compositional optimization, where nested structures complicate gradient estimation and existing methods fail to achieve optimal sample complexity. To tackle nonconvex nested stochastic optimization, this work proposes an inter-level function value tracking strategy leveraging momentum mechanisms and mini-batches to construct an efficient gradient estimator. Furthermore, a batch-free algorithm integrating first-order approximation with clipping techniques is developed, achieving comparable convergence rates without requiring average smoothness assumptions. Theoretically, the proposed method attains the optimal sample complexity of O(ε⁻⁴). Its effectiveness and superiority are empirically validated on risk-averse portfolio optimization and bilevel tilted empirical risk minimization tasks.

Abstract

This paper investigates stochastic multi-level optimization where the objective is a nested composition of several smooth non-convex functions. We assume that only stochastic estimates of the gradient and function values for each level are accessible. Consequently, obtaining an accurate estimate of the overall gradient is challenging due to the nested structure. To address this, we employ a momentum-based estimator with mini-batches to track the function values of each level, which are subsequently used to construct momentum gradient estimators. We establish an optimal sample complexity of $\mathcal{O}(ε^{-4})$ for finding an $ε$-stationary point, avoiding the stronger average smoothness assumption commonly relied upon in prior literature. Furthermore, by employing a normalization technique, we attain the same rate without requiring problem-dependent constants to set hyperparameters. To achieve the optimal rate without mini-batches, we further develop a batch-free method that incorporates a first-order approximation and a clipping technique for function value estimation. Finally, we validate the effectiveness of our proposed methods through experiments on risk-averse portfolio optimization and hierarchical tilted empirical risk minimization.

Problem

• stochastic multilevel optimization
• nested composition
• non-convex functions
• gradient estimation
• sample complexity

Innovation

• Stochastic Multilevel Compositional Optimization
• Momentum Estimator
• Sample Complexity
• Normalization Technique
• Batch-free Method