Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agents

2026-05-27

AI Summary

While external retrieval enhances the capabilities of large language model agents, it concurrently undermines their safety alignment by increasing compliance with harmful requests. This work proposes AgentREVEAL, a diagnostic framework that systematically investigates this safety degradation along two dimensions: retrieval integration mechanisms and content attributes. The analysis reveals a “safe-source paradox,” wherein even safety-oriented webpages can amplify harmful outputs, and demonstrates that single-step binding between retrieval and generation exacerbates this effect. Relevance is identified as a common trigger for both vulnerability types, highlighting an inherent trade-off between utility and safety. Evaluation on HarmURLBench—a benchmark comprising 1,405 real-world URLs spanning 320 harmful behavior categories—shows that exposure to safety-focused webpages containing warnings still increases harmful compliance rates by an average of 25%.

Abstract

AI agents augment large language models with external tools such as web retrieval, enabling grounded and up-to-date responses. However, incorporating external content into the generation pipeline can weaken the safety alignment mechanisms that govern model outputs. Prior work shows that enabling retrieval in agents increases compliance with harmful requests. We introduce AgentREVEAL, a diagnostic framework for analyzing retrieval-induced safety degradation in LLM agents. The framework examines two axes: how retrieval is integrated into the agent pipeline and the properties of the retrieved content. Along the integration axis, we find that binding tool invocation and response generation in a single step amplifies harmful outputs. Along the content axis, we uncover the Safe Source Paradox: even oppositional or safety-oriented sources, such as pages containing warnings or risk disclaimers, can increase harmful compliance by an average of 25% compared to the no-retrieval baseline. Finally, we show that relevance acts as a shared activation condition for both vulnerabilities. Similar patterns appear on frontier closed models, and harmful compliance remains elevated under several representative pipeline interventions, with some agents also entering this regime under autonomous retrieval. Because relevance is also what makes retrieval useful, these results expose a safety-utility trade-off for retrieval-enabled agents. We introduce HarmURLBench, a benchmark containing 1,405 real-world URLs paired with 320 harmful behaviors to support future evaluations.

Problem

• relevance
• safety alignment
• web retrieval
• LLM agents
• harmful compliance

Innovation

• retrieval-induced safety degradation
• AgentREVEAL
• Safe Source Paradox
• relevance-as-vulnerability
• HarmURLBench