🤖 AI Summary
In 16-bit (FP16) training, the Adam optimizer suffers from numerical instability primarily due to its sensitivity to the epsilon hyperparameter—a previously unrecognized root cause of Adam’s failure under low-precision arithmetic.
Method: We propose a lightweight, adaptive epsilon dynamic calibration mechanism that requires no additional computation, model modifications, or overhead. It adjusts epsilon in real time based solely on gradient and second-moment statistics, integrating FP16 numerical analysis, gradient scaling, and adaptive stability control while preserving the original Adam framework.
Contribution/Results: Our method significantly enhances optimization robustness in FP16 training. Experiments across multiple mainstream models demonstrate convergence speed and final accuracy on par with FP32 training, substantially improved training stability, and zero throughput degradation—achieving high-fidelity low-precision optimization without sacrificing performance or efficiency.
📝 Abstract
In this research, we address critical concerns related to the numerical instability observed in 16-bit computations of machine learning models. Such instability, particularly when employing popular optimization algorithms like Adam, often leads to unstable training of deep neural networks. This not only disrupts the learning process but also poses significant challenges in deploying dependable models in real-world applications. Our investigation identifies the epsilon hyperparameter as the primary source of this instability. A nuanced exploration reveals that subtle adjustments to epsilon within 16-bit computations can enhance the numerical stability of Adam, enabling more stable training of 16-bit neural networks. We propose a novel, dependable approach that leverages updates from the Adam optimizer to bolster the stability of the learning process. Our contributions provide deeper insights into optimization challenges in low-precision computations and offer solutions to ensure the stability of deep neural network training, paving the way for their dependable use in various applications.