SIREN (Luring LLMs onto the Rocks): PAIR-Driven Preference Manipulation in Web-RAG Recommenders

2026-07-23

AI Summary

This study addresses the problem of adversarially manipulating the recommendation rankings of web-augmented large language models (LLMs) by editing retrieved web content while keeping the retrieval source fixed. To this end, the authors propose SIREN, a method that adapts the PAIR jailbreaking framework to competitive ranking manipulation tasks by integrating an automated attacker-judge mechanism. SIREN iteratively applies 23 interpretable content poisoning techniques to elevate a target entity’s ranking without altering the retrieval source. This work presents the first demonstration of controlled competitive ranking manipulation in production-grade LLMs, introducing an interpretable taxonomy of content poisoning strategies and a custom RAG replay platform to isolate the effects of content modifications. In 124 experiments across two production Claude models, SIREN successfully ranked the target entity first in 62 cases, achieving an average replication success rate of 0.805 in new sessions, with declarative ranking claims and embedded list insertion emerging as the most effective tactics.

Abstract

This paper investigates the adversarial manipulation of the ranked recommendations produced by web-augmented large language models (LLMs). When an LLM answers a recommendation query by retrieving and reading live webpages, it acts as a recommender, and each retrieved page becomes a potential attack surface. Prior work has examined fabricated products, retrieval poisoning, and rank promotion. However, these studies do not compare how different edits to an already retrieved page change the model's final ranking while the surrounding source set remains unchanged. To address this gap, we propose SIREN, an automated attacker--judge method that adapts the PAIR jailbreaking loop to competitive rank manipulation, with the goal of moving a chosen entity to rank~1 in an LLM-generated recommendation. SIREN retrieves and captures webpages using Anthropic's web tools, then iteratively edits a retrieved source using an interpretable taxonomy of 23 content-poisoning techniques. The custom-RAG replay platform keeps the same sources in the same order, so changes in the model's ranking can be linked to changes in the supplied content rather than to differences in retrieval. Across two production Claude models, SIREN reaches rank~1 in 62 of 124 technique trials nested within eight query--model contexts. The payloads that reached rank~1 were then tested in fresh sessions, where they reproduced the result with a mean success rate of 0.805. Across the evaluated settings, declarative ranking claims and seeded lists were generally more effective than directive-form injections, although the strength of this difference depended on the target model. To the best of our knowledge, this is among the first controlled studies of competitive rank manipulation in production LLMs where the supplied source context is kept fixed.

Problem

• adversarial manipulation
• rank manipulation
• web-augmented LLMs
• recommendation systems
• content poisoning

Innovation

• SIREN
• Preference Manipulation
• Web-RAG
• Content Poisoning
• Rank Promotion