🤖 AI Summary
This study addresses the problem of burst-induced job wait time exacerbation in cross-facility scientific workflows, leveraging production logs from the Jean Zay supercomputer. Methodologically, it reveals the independence between burst penalties and Slurm’s fair-share factor, proposes a short-cycle throttling mechanism, and constructs a model capturing the strong correlation between job wait times and intra-burst rankings. These findings are quantitatively validated through Pearson correlation analysis, convolution formula derivation, and Welch’s t-tests. Results demonstrate that over 40% of jobs exhibit wait times highly correlated with their burst rankings. Furthermore, partition occupancy and requested core counts are identified as critical influencing factors. This work provides empirical evidence for optimizing scheduling fairness in high-performance computing environments.
📝 Abstract
This paper examines job waiting times on the Jean Zay supercomputer at IDRIS (Institut du D{é}veloppement et des Ressources en Informatique Scientifique http://www.idris.fr/), driven by the need to improve scheduling strategies for crossfacility scientific workflows. Drawing on large workload traces, we show that job waiting times are not independent of one another but are instead correlated through job bursts-batches of identical jobs submitted together, also called ``flurries''. We introduce a method for quantifying the effect of these bursts on waiting times, and find a strong Pearson correlation between waiting time and a job's rank within its burst. This correlation is above 0.8 for more than 40% of the jobs. We further derive an exact formula for the Slurm fairshare factor based on a convolution product, showing that the burst penalty operates on a much shorter timescale than fairshare itself-pointing to a distinct throttling mechanism specifically targeting bulk submissions. Once burst effects are accounted for, we use Welch's t-test to find that partition occupancy and number of requested cores have a significant effect on waiting time, though the effect is not significant for all partitions.