False Prophets: On the Security of World Models in Agentic Systems

2026-07-25

AI Summary

This work addresses the unique security risks posed by world models in autonomous agent systems, which adversaries may exploit to induce prediction errors leading to harmful actions or sensitive data leakage. The study presents the first systematic identification and formal definition of security vulnerabilities specific to world models, along with the creation of the first safety benchmark dataset tailored for textual world models. Through adversarial testing, vulnerability analysis, and simulated attacks—empirically evaluated within end-to-end agent environments—the authors demonstrate the practical feasibility of these threats, achieving attack success rates as high as 95% in scenarios including cryptocurrency wallet theft and extraction of confidential information. Based on these findings, the paper concludes with targeted mitigation strategies to enhance the robustness of world model–based agents.

Abstract

Large language models now power autonomous agents capable of complex, multi-step tasks in different environments. Accurate and reliable execution of these tasks requires the agent to predict the results of its actions. Recent research proposes to enhance predictive capabilities via specially trained environment simulators-world models. While world models can improve performance, they can also mislead agents into executing harmful actions, creating significant security and privacy risks. In this paper, we raise security concerns regarding the usage of world models in agentic systems. We discover a range of world model specific vulnerabilities, which can be exploited in terminal-based agents to execute malicious code or extract sensitive data. To facilitate future development, we introduce a security benchmark dataset designed for text-based world models. We argue that some risks are intrinsic to approximate world modeling, and show that attackers can induce mispredictions in agentic pipelines with up to 95% success rate, possibly resulting in unintended command execution, denial of service, drainage of wallet and private information extraction. Finally, we provide practical recommendations for practitioners to mitigate the discovered harms and harden agentic systems.

Problem

• world models
• agentic systems
• security risks
• adversarial attacks
• privacy leakage

Innovation

• world models
• agentic systems
• security vulnerabilities
• adversarial attacks
• AI safety