🤖 AI Summary
This work addresses the limitations of existing high-performance computing workload datasets, which often lack long-term temporal coverage, real memory usage traces, and diverse user origins, thereby hindering accurate evaluation of scheduling algorithms. Leveraging comprehensive job logs from the French IN2P3 Computing Center in 2024, this study presents a publicly available, high-fidelity production-grade dataset encompassing 44 million jobs submitted by 1,000 users across up to 312 machines, with concurrent execution of 46,000 threads and aggregate memory consumption of 105 TB. The dataset substantially enhances realism and timeliness, enabling in-depth analysis of user behavior patterns ranging from weekly to seasonal scales, and provides a critical foundation for more realistic simulation and performance evaluation of scheduling algorithms.
📝 Abstract
This paper provides and analyzes a dataset detailing the characteristics and execution data of all jobs submitted to the IN2P3 Computing Center (Villeurbanne, France), a national research and support unit of the CNRS, in 2024. The main additional value of this contribution compared to previously available datasets consists in the combination of an extended time interval considered, the inclusion of memory usage data and its recency, on top on improving the diversity of datasets provenance. This allows researchers to simulate and evaluate scheduling algorithms on a real workload over a large time window. Thus, specificities due to seasonal, monthly, and weekly user behaviors can be taken into account, which is not possible with smaller or synthetic datasets. It is composed of 44M jobs submitted by 1k users running on a cluster of a maximum of 312 machines supporting 46k concurrent threads and providing 105To of RAM.