Compositional Threat Analysis of Latent Compromise in LLM Agent Systems: The Order 66 Scenario

2026-08-08

AI Summary

This study addresses combinatorial security threats in large language model agents arising from the interplay of latent rules, activation signals, and high-privilege execution. It formally introduces the “Order 66” paradigm—a novel security framework wherein individually trustworthy components become hazardous when combined. The authors construct a compositional threat model to analyze interactions among latency, triggering conditions, privilege escalation, target reachability, and recovery failure, identifying three propagation pathways: pre-deployment embedding, post-deployment injection, and peer-to-peer replication. Integrating compositional modeling, propagation dynamics, and cross-category feedback loops, the work demonstrates the inadequacy of current detection mechanisms. While a complete attack chain has not yet been observed in practice, all constituent elements are shown to be realistically feasible. The paper concludes by proposing four effective defense strategies: capability mediation, state provenance, propagation isolation, and protected recovery.

Abstract

In the fictional Order 66, catastrophe does not arise from a powerful command alone: a trusted population is preconditioned, a short directive activates the concealed condition, and protective authority turns against the system. This paper translates that mechanism into an origin-neutral security analysis of tool-using large language model (LLM) agents. A representative scenario combines a deployed artifact or shared memory bearing a dormant destructive rule, a later email, document, update, or peer message that activates it, and an agent harness granting operational and recovery authority. We introduce a compositional model explaining why no component is catastrophic alone, yet their conjunction can produce correlated destructive action. We separate three population-reach routes --- release-time pre-positioning, post-release durable seeding, and peer replication --- from a common core of dormancy, activation, authority, reachable targets, and failed recovery. This yields defensive cut sets and shows why checkpoint scanning or prompt filtering cannot close every route. A two-class example shows that cross-class feedback can sustain spread even when both within-class reproduction terms are below one; isolation and persistence controls suppress the loop. Published work instantiates constituent mechanisms, while incidents demonstrate autonomous boundary crossing, malicious agent extensions, agent-assisted reconnaissance, and public-package propagation, but not the full dormant-implant composition. We found no public observation, in evidence reviewed through 5 August 2026, traversing the complete Order 66 graph. The result is neither dismissal nor prediction: the scenario is componentwise credible under stated assumptions, damage depends on the harness, and the strongest defenses are capability mediation, durable-state provenance, propagation isolation, and protected recovery.

Problem

• latent compromise
• LLM agent systems
• compositional threat
• dormant implant
• Order 66 scenario

Innovation

• compositional threat analysis
• latent compromise
• LLM agent security
• dormant implant
• propagation isolation