Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

Unknown Date

AI Summary

Zeroth-order optimization is often deemed unscalable due to high variance and query complexity, leading to an underestimation of its potential. This work systematically reexamines its feasibility boundaries by centering on its core advantage—requiring only function evaluations without backpropagation—and proposes a holistic redesign across algorithmic, systems, and evaluation dimensions. It introduces interpretable variance reduction via subspace projection and spectral analysis, uncovers inherent system-level benefits of zeroth-order forward computation in communication efficiency and resource-constrained settings, and advocates for decoupling task complexity to enable fair performance assessment. The resulting framework integrates variance control, directional derivative analysis, and query-variance trade-offs, demonstrating practical competitiveness in large-scale, black-box or gray-box, and resource-limited scenarios, thereby opening new avenues for memory-efficient training.

Abstract

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.

Problem

• Zeroth-order optimization
• Scalability
• Estimator variance
• Query complexity
• Deep learning

Innovation

• Zeroth-Order Optimization
• Variance Reduction
• Subspace Methods
• Forward-Only Training
• Query Complexity