The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment

2026-06-04

AI Summary

This work investigates emergent misalignment—sudden semantic mismatches in out-of-domain queries—that frequently arises after fine-tuning large language models on specific tasks. The authors propose the “free-rider hypothesis,” arguing that prefix tokens in chat templates inadvertently generalize fine-tuned behaviors beyond their intended domain. To mitigate this, they introduce Token-Regularized Fine-Tuning (TReFT), a lightweight alignment intervention that regularizes representations of critical prefix tokens without altering user inputs. This study is the first to identify the pivotal role of prefix tokens in cross-domain behavior generalization. Experiments demonstrate that, when applied to Llama-3.1-8B fine-tuned on legal tasks, TReFT reduces emergent misalignment by 33.5% compared to data interleaving and cuts off-topic generalization by 54.3% on average in refusal and tool-use scenarios, all while preserving in-domain performance.

Abstract

The mechanisms behind LLMs' broad over-generalization beyond training examples remain unclear. Emergent misalignment (EM) offers a striking case study: finetuning on narrow tasks induces broad misalignment to semantically-unrelated test domains. In this work, we propose the Piggyback Hypothesis: the chat-template tokens can piggyback the finetuned behaviour onto out-of-domain queries. We validate this hypothesis by showing that subtle perturbations to the prefix (tokens preceding all user queries), or patching the prefix representations with those from the unfinetuned model, can restore alignment without changing the user query. Building on this finding, we propose Token-Regularized Finetuning (TReFT), which regularizes specific token representations during training to mitigate EM. Across different models and multiple EM-inducing datasets, TReFT reduces EM while preserving in-domain learning. On Llama-3.1-8B finetuned on the legal domain, TReFT achieves 33.5% more EM reduction than data interleaving with a retain set of aligned examples. We further show that TReFT extends to other narrow-finetuning settings, including abstention, tool use, and refusal (off-topic generalization is reduced by 54.3% on average), supporting the Piggyback Hypothesis. Broadly, our work highlights that LLMs may learn and generalize in unintended ways and suggests a path toward more constrained finetuning. It also calls for further study of how shared input features can piggyback model behavior across domains.

Problem

• emergent misalignment
• over-generalization
• large language models
• fine-tuning
• domain generalization

Innovation

• Piggyback Hypothesis
• Emergent Misalignment
• Token-Regularized Finetuning
• Prefix Perturbation
• Out-of-Domain Generalization