Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness

2026-09-30

AI Summary

This study addresses the phenomenon whereby alignment training causes large language models to silently deviate from inputs, creating a conflict between alignment and faithfulness. We formally define "Alignment-Induced Unfaithfulness" (AIU), propose a capability-alignment-faithfulness trilemma, and reveal an inverse scaling law wherein AIU intensifies with model size. To systematically investigate this, we construct the FaithConflict dataset alongside a dual taxonomy encompassing behavioral and chain-of-thought dimensions, analyzing the mechanisms of post-training methods such as DPO. Our results demonstrate that AIU is covertly amplified at intermediate checkpoints and remains resistant to prompt-based mitigation, confirming its deep overriding effect on task inputs. Ultimately, this work establishes faithfulness as a critical new constraint for aligned model design.

Abstract

Large language models are characterized by three key properties: capability, alignment, and faithfulness. Prior work studies the tradeoffs between capability and alignment, and between capability and faithfulness, but a third tension remains underexplored: the alignment-faithfulness conflict. We show that aligned models systematically deviate from their inputs on unsafe or sensitive content without disclosing the modification, a failure mode we call alignment-induced unfaithfulness (AIU). Unlike capability-driven unfaithfulness, which comes from errors in knowledge or reasoning, this is induced by post-training mechanisms that override adherence to the input. We introduce FaithConflict, a controlled dataset isolating both conflicts, and two complementary taxonomies: behavioral (B1-B8) and chain-of-thought reasoning (C0-C6). Across models, AIU increases with scale and more sharply than capability-driven unfaithfulness, a reverse scaling law; intermediate checkpoints show it is amplified during post-training, with DPO the stage at which the gap both grows most and becomes least visible. Prompting-based mitigation does not resolve it, revealing a capability-alignment-faithfulness trilemma in the design and evaluation of LLMs.

Problem

• Alignment-Faithfulness Conflict
• Alignment-Induced Unfaithfulness
• Large Language Models
• Post-training
• Trilemma

Innovation

• Alignment-induced Unfaithfulness
• FaithConflict Dataset
• Reverse Scaling Law
• Capability-Alignment-Faithfulness Trilemma
• Direct Preference Optimization