Nix to the Rescue for a Reproducible HPC-AI Software Stack

2026-07-29

AI Summary

This work addresses the challenge of constructing reproducible software stacks in high-performance computing (HPC) and AI convergence scenarios, where constraints such as lack of root privileges, network isolation, and heterogeneous language environments hinder conventional tooling. For the first time, it systematically applies Nix’s fully isolated build model together with its declarative flake configuration system to HPC-AI hybrid environments, enabling unified management of C/C++ and Python dependencies. By automatically generating Apptainer containers, the approach ensures consistency between local development and deployment on supercomputing systems. The method effectively resolves critical issues including dependency discovery, system library leakage, and cross-project composition, achieving highly reproducible deployments across non-root workstations and production clusters. It demonstrates clear advantages over traditional module systems, Conda environments, and manual containerization, while also highlighting current gaps in Nixpkgs’ coverage of machine learning packages.

Abstract

Reproducibility in HPC remains difficult under the constraints of production supercomputers: no root access, limited internet, and software stacks that increasingly span C/C++, Fortran, Python, MPI, and GPU runtimes. Traditional approaches based on environment modules and Conda require manual intervention to locate dependencies, leak system libraries into builds, and fail to compose across projects. Containers help with deployment but do not by themselves guarantee reproducibility. We report on our experience building a hybrid HPC/AI software stack with Nix, covering local development on a workstation without root and remote deployment as an Apptainer image on a production cluster. Nix's consistent package layout, full environment isolation, and flake-based composition resolve the dependency discovery, leakage, and composition problems we encountered, while unifying C/C++ and Python management under a single declarative specification that also generates the deployment container. We discuss trade-offs against Spack and Guix, the development-versus-production split addressed via CMake presets, and current gaps in ML package coverage in Nixpkgs.

Problem

• reproducibility
• HPC-AI software stack
• dependency management
• environment isolation
• supercomputing constraints

Innovation

• Nix
• reproducibility
• HPC-AI software stack
• environment isolation
• declarative packaging