Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning

2026-07-25

AI Summary

This work addresses the vulnerability of large language models to subtle data contamination during fine-tuning, where seemingly benign or minimal poisoned examples can induce latent harmful behaviors that evade existing defenses. To mitigate this risk, the authors propose a consensus decoding mechanism at inference time, leveraging multiple independently fine-tuned reference models and aggregating their output distributions to retain only behaviors consistently exhibited across all models. The approach employs token-wise minimum probability and base-model-relative consensus strategies, enabling flexible yet robust alignment with shared intent while circumventing limitations of training-time defenses. Experimental results demonstrate that this method effectively suppresses source-specific harmful behaviors induced by data poisoning, subconscious learning, and emergent misalignment, while preserving beneficial capabilities common across models, significantly outperforming baselines such as joint training and weight averaging.

Abstract

Recent work shows that fine-tuning language models on even a small amount of poisoned data can install targeted misbehavior, and ostensibly benign data can transmit hidden preferences that generalize broadly. Standard defenses, such as data filtering, mixing in harmless data, and regularization, attenuate these effects but do not eliminate them. We instead pursue robustness through redundancy: collecting multiple datasets from different sources and only learning what is common between them. Thus, if only a subset of sources are malicious, the misbehavior will be blocked. In order to implement this defense strategy, we fine-tune a separate reference model on each source's dataset and aggregate their next-token distributions at decoding time. We introduce two consensus decoders: a token-wise minimum, which caps each token at the lowest probability any source assigns, and a base-relative variant, which reverts to the base probability on any token the sources move in opposing directions. We further relax exact agreement to tolerate partial support across sources and different surface expressions of the same intention. Across controlled poisoning tasks, subliminal learning, and emergent misalignment, consensus decoding suppresses source-specific misbehavior while preserving shared desirable behavior, including cases where union training and weight averaging retain the unwanted behavior.

Problem

• LLM fine-tuning
• hidden behaviors
• data poisoning
• misalignment
• adversarial robustness

Innovation

• inference-time consensus
• consensus decoding
• LLM fine-tuning robustness
• hidden behavior mitigation
• multi-source alignment