Problem
Research questions and friction points this paper is trying to address.
Workflow portability
High-performance computing
Site profiles
Automated deployment
Resource configuration
The Other Half of Workflow Portability: Evidence-Backed HPC Site Profiles with Agentic Discovery
🤖 AI Summary
This study addresses the reliance on manual configuration when deploying high-performance computing (HPC) workflows across heterogeneous sites by proposing an evidence-based method for the automated generation of structured site configurations. The approach leverages constrained language model agents to extract relevant information from documentation, combined with automated node profiling and pilot job execution for rule validation, thereby bridging the critical knowledge gap concerning site-specific usage practices in workflow portability. Experimental evaluations conducted at Purdue, TACC, and Notre Dame demonstrate that the proposed method successfully constructs accurate configuration files, enabling automated pre-flight verification of real-world workflows while providing interpretable failure diagnostics.
📝 Abstract
Moving a workflow developed and tested at one HPC site to another rarely succeeds without some amount of trial and error. Package managers rebuild software environments, containers ship whole filesystems, and workflow specifications such as backpacks package a workflow with its software, data, and resource requirements. These approaches address one half of workflow portability: what a workflow needs. But none describes how a given HPC site must be used, and that missing half is why even a portable workflow requires manual adjustment at each new site. That gap includes the site's resource shape, storage configuration, network permissions, and operating policies. This information may be explicit in the batch system, hidden in the prose of documentation, or buried deep within a router's configuration, making it difficult for an automated deployment tool to turn site knowledge into useful deployment decisions. We propose the HPC site profile, a structured, evidence-backed document that makes this knowledge actionable. We automatically construct it in three steps that mirror where the information lives: measuring the login node, extracting typed fields from documentation with a bounded language-model agent, and submitting pilot jobs for eligible unresolved fields. Every field is verified against its evidence or discarded, so a rule, not the model, decides what enters the profile. The profile then preflights a workflow into an execution plan or an early, explainable failure. We build profiles at Purdue Anvil, TACC Stampede3, and Notre Dame CRC and present a case study of preflighting a real workflow.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Workflow Portability
HPC Site Profile
Agentic Discovery
Language-Model Agent
Preflighting
🔎 Similar Papers
No similar papers found.
