🤖 AI Summary
This study addresses the prevalent yet elusive issue of “silent failures” in AI-generated code—functional degradations or errors that appear syntactically correct but evade detection by conventional testing methods. The authors propose the Reward Shaping Failure Hypothesis, attributing this phenomenon to optimization dynamics in human feedback alignment, and introduce the novel concept of “failure truthfulness.” Building on this insight, they develop AIRA, the first targeted risk auditing framework for AI-generated code, which employs structured static analysis with 15 deterministic rules across JavaScript, Python, and TypeScript, with particular emphasis on exception handling. Empirical evaluation reveals that AI-generated code exhibits 1.80 times more high-severity issues per file (0.435) than human-written code (0.242), a consistent effect across languages, with deficiencies in exception handling being especially pronounced.
📝 Abstract
Practitioners have reported a directional pattern in AI-assisted code generation: AI-generated code tends to fail quietly, preserving the appearance of functionality while degrading or concealing guarantees. This paper introduces the Reward-Shaped Failure Hypothesis - the proposal that this pattern may reflect an artifact of optimization through human feedback rather than a random distribution of bugs. We define failure truthfulness as the property that a system's observable outputs accurately represent its internal success or failure state. We then present AIRA (AI-Induced Risk Audit), a deterministic 15-check inspection framework designed to detect failure-untruthful patterns in code. We report results from three studies: (1) an anonymized enterprise environment audit, (2) a balanced 600-file public corpus pilot, and (3) a strict matched-control replication comparing 955 AI-attributed files against 955 human-control files. In the final replication, AI-attributed files show 0.435 high-severity findings per file versus 0.242 in human controls (1.80x). The effect is consistent across JavaScript, Python, and TypeScript, with strongest concentration in exception-handling-related patterns. These findings are consistent with a directional skew toward fail-soft behavior in AI-assisted code. AIRA is designed for governance, compliance, and safety-critical systems where fail-closed behavior is required.