🤖 AI Summary
This study addresses the uncertain practical efficacy of dynamic resource management—particularly MPI variability—in high-performance computing (HPC) environments. To bridge this gap, the authors propose a methodology based on replaying real job logs to faithfully reproduce actual workloads on a 125-node partition of the MareNostrum 5 supercomputer, thereby offering the first validation of MPI variability in a real-world HPC setting. By introducing a parallel-efficiency-oriented variability strategy integrated with resource scheduling and performance monitoring mechanisms, the experiments demonstrate a 27% reduction in execution time for variable workloads without compromising baseline job performance or overall resource utilization. These results provide compelling evidence of the approach’s effectiveness and practical value in authentic user scenarios.
📝 Abstract
Dynamic Resource Management (DRM) techniques can be leveraged to maximize throughput and resource utilization in computational clusters. Although DRM has been extensively studied through analytical workloads and simulations, skepticism persists among end administrators and users regarding their feasibility under real-world conditions. To address this problem, we propose a novel methodology for validating DRM techniques, such as malleability, in realistic scenarios that reproduce actual cluster conditions of jobs and users by replaying workload logs on a High-performance Computing (HPC) infrastructure. Our methodology is capable of adapting the workload to the target cluster. We evaluate our methodology in a malleability-enabled 125-node partition of the Marenostrum 5 supercomputer. Our results validate the proposed method and assess the benefits of MPI malleability on a novel use case of a pioneer user of malleability (our "PhD Student"): parallel efficiency-aware malleability reduced a malleable workload time by 27% without delaying the baseline workload, although introducing queueing delays for individual jobs, but maintaining the resource utilization rate.