Covert Influence Between Language Models

2026-06-02

AI Summary

This work demonstrates that language models can covertly propagate behavioral tendencies through natural language prompts that are imperceptible to humans, posing a novel security risk across interfaces such as supervised fine-tuning, policy distillation, and in-context learning. Unlike prior approaches relying on numerical or synthetic triggers, this stealthy influence exhibits stronger human undetectability and cross-model-family portability. To address this, the authors introduce a sample-wise attribution scoring method at inference time that effectively identifies and amplifies such hidden influences. Experiments for the first time achieve highly covert behavioral transfer, revealing that the scope of this risk substantially exceeds current understanding, and validate the proposed attribution technique as an effective tool for both detection and mitigation.

Abstract

As language models increasingly consume one another's outputs, covert influence -- a phenomenon where a sender's payload (the behavioral disposition it is conditioned to propagate) transfers to a receiver through carriers undetectable by humans -- becomes a growing risk. We characterize this risk across three interfaces: supervised fine-tuning, on-policy distillation, and in-context learning, and find that they vary in the scale of influence achievable without leaving behind human-visible traces. Using inference-time per-sample attribution scores, we study covert influence across all three interfaces with the ability to select carriers that amplify training-time influence, unlocking payload transfers that prior work could not achieve. We further provide evidence that covert influence with natural-language carriers is a distinct phenomenon from prior studies using number carriers, as the latter is more resistant to human detection and less portable across model families. Together, these results suggest that the risk surface for covert influence is broader than previously recognized, and we study pointwise attribution scoring methods as a tool to investigate and mitigate it.

Problem

• covert influence
• language models
• behavioral payload
• undetectable carriers
• model interaction

Innovation

• covert influence
• attribution scoring
• natural-language carriers
• model distillation
• in-context learning