PRWeaver: Evaluating LLM-Based Code Auditors against Long-Horizon Malicious Pull Requests

2026-08-03

AI Summary

This study systematically evaluates the capability of large language models (LLMs) to detect stealthy, cross-commit malicious pull requests (PRs) that evolve over extended periods within code review workflows. To this end, the authors introduce PRWeaver, a benchmark comprising 208 execution-verified attack samples from 10 real-world repositories, along with novel attack techniques such as context weaving and payload fusion, and multi-granularity PR presentation strategies. Experimental results demonstrate that under a full historical window (N=24), state-of-the-art LLM-based auditing systems achieve detection rates of only 16–22%, substantially lower than the 50–60% attainable through per-PR inspection. Context weaving and payload fusion reduce detection performance by 5–13 and 10–18 percentage points, respectively, revealing for the first time that reliance solely on repository history is insufficient to defend against semantically obfuscated attacks.

Abstract

LLM-based code auditors are increasingly integrated into pull-request (PR) workflows, yet their reliability against adversarial changes distributed across repository evolution remains poorly understood. We introduce PRWeaver, a benchmark of 208 execution-validated attacks from ten real-world repositories, each instantiated under four matched review renderings (832 renderings in total). We evaluate three PR-auditing agents across six auditor-model systems. Across all systems, decomposing an attack changes detection by at most five percentage points, showing that commit boundaries alone do not explain evasion. In contrast, per-PR interleaving at $N=16$ and coherent carrier fusion reduce detection by 5-13 and 10-18 points, respectively. Under whole-window review at $N=24$, detection falls to 16-22%, compared with 50-60% under per-PR review. These results show that access to repository history is insufficient: concealment becomes most effective when benign and malicious changes jointly occupy the auditor's active review context or when the stated purpose plausibly accounts for the attack-bearing diff.

Problem

• LLM-based code auditing
• malicious pull requests
• long-horizon attacks
• code review evasion
• repository evolution

Innovation

• PRWeaver
• LLM-based code auditing
• long-horizon adversarial attacks
• coherent carrier fusion
• review context interleaving