Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning

2026-05-31

AI Summary

This work addresses the vulnerability of supervised fine-tuning to out-of-distribution performance collapse, a failure mode poorly anticipated by conventional training signals. The study identifies that such risks strongly correlate with representational drift along specific low-dimensional directions within the model’s internal activations. Building on this insight, the authors propose an efficient monitoring mechanism that aligns with these critical feature directions. By integrating linear probing, LoRA fine-tuning, PCA, and sparse autoencoders, the method achieves exceptional detection performance—0.990 AUROC, 2.2% false negative rate, and 2.9% false positive rate—across four open-source large language models (7–9B parameters). It substantially outperforms unsupervised baselines and demonstrates robustness under stress tests across diverse models and training configurations.

Abstract

Emergent misalignment (EM) occurs when narrow finetuning causes a model to behave dangerously outside the finetuning task. Standard training signals can miss this shift, making reliable detection costly if it depends on repeated behavioral evaluation. We ask whether emergent misalignment can instead be detected from internal representations during finetuning. Using seven alignment-relevant traits encoded as linear directions in activation space, we track representational drift across training checkpoints in four open-source 7-9B LLMs. EM-relevant drift concentrates on a low-dimensional axis that explains 65.5% of the variance, revealing a geometric signature in the studied regime. A low-overhead monitor built on this drift profile detects dangerous checkpoints with 2.2% false negative rate, 2.9% false positive rate, and 0.990 AUROC on held-out perturbation types, outperforming unsupervised PCA and SAE baselines. Stress tests on two 14B models, longer finetuning runs, and misaligned starting points identify key deployment boundaries. These results position trait-space monitoring as a practical complement to behavioral evaluation for EM detection during LoRA-based finetuning, while showing that deployment across substantially different regimes may require recalibration.

Problem

• emergent misalignment
• supervised finetuning
• trait-space monitoring
• representational drift
• LLM safety

Innovation

• trait-space monitoring
• emergent misalignment
• representational drift
• low-dimensional signature
• LoRA finetuning