🤖 AI Summary
This work addresses two critical limitations of Git-based data versioning tools (e.g., DataLad) in HPC environments: incompatibility with the Slurm batch scheduler and poor I/O efficiency on parallel file systems (e.g., Lustre, GPFS). We propose the first lightweight, non-intrusive framework that deeply integrates DataLad with Slurm. Our approach extends Slurm’s job encapsulation mechanism and introduces automated, fine-grained metadata capture at the job level, thereby ensuring end-to-end reproducibility. Furthermore, we optimize versioning operations—such as dataset checkout and commit—by adapting their I/O paths for parallel file systems. Evaluation on a production supercomputing cluster demonstrates a 92% reduction in metadata capture overhead and a 3.8× speedup in large-dataset version switching. These improvements significantly broaden the applicability of data version control to production-scale HPC batch workflows.
📝 Abstract
We present the adaptation of an existing data versioning and machine-actionable reproducibility solution for HPC. Both aspects are important for research data management and the DataLad tool provides both based on the very prevalent git version control system. However, it is incompatible with HPC batch processing. The presented extension enables DataLad's versioning and reproducibility in conjunction with the HPC batch scheduling system Slurm. It solves a fundamental incompatibility as well as inefficient behavior patterns on parallel file systems.