Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems

2024-12-03

arXiv.org

AI Summary

Addressing the hyperparameter optimization (HPO) challenge for TB-scale scientific datasets in high-performance computing (HPC) environments—where conventional methods are computationally prohibitive due to insufficient resources to complete full training iterations—this work proposes a resource-adaptive successive halving mechanism. It treats GPU count as a dynamically schedulable resource, integrating asynchronous successive halving (ASHA), data-parallel training, dynamic batch-size scheduling, and coordinated large-scale (up to 1,024 GPUs) orchestration. To our knowledge, this is the first systematic and scalable HPO framework demonstrated on TB-scale scientific data. Compared to standard ASHA, it achieves a 1.9× speedup while preserving or improving final model accuracy. The methodology has been successfully deployed across three large-scale application domains: computer vision, computational fluid dynamics, and additive manufacturing, enabling efficient neural network training at unprecedented data and hardware scales.

Technology Category

• Machine Learning: Hardware-aware ML
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

On High-Performance Computing (HPC) systems, several hyperparameter configurations can be evaluated in parallel to speed up the Hyperparameter Optimization (HPO) process. State-of-the-art HPO methods follow a bandit-based approach and build on top of successive halving, where the final performance of a combination is estimated based on a lower than fully trained fidelity performance metric and more promising combinations are assigned more resources over time. Frequently, the number of epochs is treated as a resource, letting more promising combinations train longer. Another option is to use the number of workers as a resource and directly allocate more workers to more promising configurations via data-parallel training. This article proposes a novel Resource-Adaptive Successive Doubling Algorithm (RASDA), which combines a resource-adaptive successive doubling scheme with the plain Asynchronous Successive Halving Algorithm (ASHA). Scalability of this approach is shown on up to 1,024 Graphics Processing Units (GPUs) on modern HPC systems. It is applied to different types of Neural Networks (NNs) and trained on large datasets from the Computer Vision (CV), Computational Fluid Dynamics (CFD), and Additive Manufacturing (AM) domains, where performing more than one full training run is usually infeasible. Empirical results show that RASDA outperforms ASHA by a factor of up to 1.9 with respect to the runtime. At the same time, the solution quality of final ASHA models is maintained or even surpassed by the implicit batch size scheduling of RASDA. With RASDA, systematic HPO is applied to a terabyte-scale scientific dataset for the first time in the literature, enabling efficient optimization of complex models on massive scientific data. The implementation of RASDA is available on https://github.com/olympiquemarcel/rasda

Problem

• Optimizing hyperparameters for large datasets on HPC systems efficiently
• Reducing runtime while maintaining model quality in hyperparameter optimization
• Enabling systematic hyperparameter tuning for terabyte-scale scientific datasets

Innovation

• Combines resource-adaptive successive doubling with asynchronous halving
• Uses implicit batch size scheduling for neural networks
• Enables hyperparameter optimization on terabyte-scale scientific datasets