🤖 AI Summary
This work addresses the challenges posed by dynamic AI systems, where models, input distributions, evaluation datasets, and objectives continually evolve, rendering traditional static configurations inadequate. To tackle this, the study proposes integrating tunable elements—such as model selection and prompt structures—into software engineering governance, establishing an engineering paradigm centered on versioned and controlled program spaces. By leveraging statistical methods including sampled evaluation sets, effect size bounds, and confidence/risk thresholds, the framework enables evidence-based mechanisms for updating and deploying system variables. This approach facilitates ongoing validation of relative system effectiveness under shifting conditions, thereby fostering AI engineering practices that are maintainable, verifiable, and capable of sustainable evolution, significantly enhancing system robustness and adaptability.
📝 Abstract
The maintained artifact in an AI-enabled system is not code plus settings, but a versioned governed program space: domains, structural constraints, eligibility, evaluation assets, and a statistical release gate. AI-enabled systems operate under changing world conditions: provider models and APIs change, input distributions drift, evaluation sets age, and objectives such as quality, cost, latency, and safety are renegotiated over time. In practice, teams often respond through ad hoc changes to model choice, retrieval policy, prompt structure, and operational thresholds. Fixed-assignment reasoning is therefore insufficient: a chosen assignment is valid only relative to an environment, evaluation set, and policy state. We argue that such choices should be treated as tuned variables: program variables maintained under governance as environments and evaluation sets evolve. Building on SE4AI work and our prior work on governed tuning, this paper positions the governed space as the software-engineering object. Here, statistical means that promotion relies on sampled evaluation sets, estimated evidence, effect-size margins, and confidence/risk thresholds.