Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation

2026-07-31

AI Summary

This study systematically investigates the nonlinear degradation of instruction-following performance in large language models when confronted with multiple stacked instructions. To this end, the authors construct a benchmark dataset comprising up to 20 verifiable instructions per prompt and develop an automated verifier alongside cluster-robust statistical testing methods. They uncover, for the first time, a structural conflict mechanism induced by instruction stacking and propose a training-free prompt compiler that rewrites prompts in a single invocation to mitigate such conflicts. Experimental results demonstrate that instruction adherence drops from approximately 96% to as low as 20% as the number of instructions increases. The proposed compiler yields gains of up to 11 percentage points for weaker models, while exerting minimal impact on stronger ones.

Abstract

Production prompts rarely carry a single instruction. One system message may require valid JSON, a word limit, three citations, and a fixed tone at the same time. We study how instruction-following degrades as such constraints accumulate. We introduce a benchmark that stacks 24 verifier-checked instructions, one to twenty at a time, and evaluate three production-tier LLMs (Claude Sonnet 4.6, GPT-5-mini, Gemini 2.5 Flash). Instruction-following degrades non-linearly: the follow rate falls from ~96% to as low as 20%, driven by a structured and reproducible set of pairwise conflicts. A single "output JSON" constraint, for example, is jointly unsatisfiable with nine others. We then evaluate a training-free remedy: an instruction compiler that rewrites the stacked prompt in a single LLM call and is reused across queries. Its benefit is capability-graded. It recovers up to +11 points of follow rate for weaker models, which are also the models most often deployed at scale, while leaving stronger models, which already internalise the same structure, essentially unchanged. Cluster-robust tests, same-baseline controls, and a within-family scaling ladder attribute the gain to the rewrite itself rather than to additional tokens, reordering, or measurement headroom. We release the benchmark, verifiers, and cached runs for full reproduction.

Problem

• instruction stacking
• instruction-following degradation
• prompt constraints
• LLM benchmark
• verifier-checked instructions

Innovation

• instruction stacking
• prompt compilation
• instruction-following degradation
• verifier-based benchmark
• training-free remedy