Cheap, open agents make LLM pollution harder to mitigate

2026-09-25

AI Summary

This study addresses the escalating risk of large language model data contamination posed by low-cost, open-source agents, which evade conventional detection methods. Through survey-based experiments across nine agent configurations utilizing open-weight models and agent frameworks, this work systematically compares the response characteristics and detectability of commercial versus open-source systems. The findings demonstrate for the first time that fully local, open-source agents achieve commercial-grade competitiveness, and that no single metric reliably distinguishes all agents, with open-text analysis exhibiting the highest discriminative power. Accordingly, this paper proposes a multi-dimensional, automated, multi-level detection strategy, establishing a defensive direction against emerging forms of data contamination.

Abstract

Large Language Model (LLM) pollution occurs when synthetic responses contaminate data intended to capture human behavior. High deployment costs have so far limited the risk posed by autonomous survey agents. However, open-weight models paired with open-source agentic frameworks may have removed this barrier. We compared the performance and detectability of nine agent configurations, ranging from fully open variants to closed commercial ones. Each agent autonomously completed a survey containing multiple response types yielding various detection checks. Fully open agents ran locally without usage fees and performed competitively with commercial alternatives. Open and commercial agents failed different sets of checks, and no single check reliably detected all agents, but open-text responses discriminated best between agents and humans. These findings identify fully open agents as a distinct risk for LLM pollution and support multilayered detection strategies emphasizing open-text analysis.

Problem

• LLM pollution
• open agents
• autonomous survey agents
• synthetic data contamination
• detectability

Innovation

• LLM pollution
• open-weight models
• autonomous survey agents
• multi-layered detection
• open-text analysis