Constitutional Midtraining: Content Presence Drives Alignment Gains

2026-07-29

AI Summary

This work addresses the fragility of existing post-training alignment methods, which often degrade during subsequent fine-tuning due to a lack of persistence guarantees. To enhance alignment robustness, the authors propose embedding principle-based constitutional corpora during mid-training and systematically investigate—via a 2×2 factorial experiment—the impact of curriculum sequencing and reasoning strategies on alignment outcomes. Leveraging a 120B-parameter model, 394M tokens of constitutional text, and a structured curriculum design, combined with a multi-stage evaluation framework, the study demonstrates for the first time that mid-training constitutional intervention significantly improves alignment persistence. Specifically, it reduces undesirable behaviors in ransomware scenarios by 17.5 percentage points over baselines—a gain retained even after subsequent benign fine-tuning—without compromising performance on general benchmarks such as MMLU and GSM8K.

Abstract

Post-training alignment is often shallow, eroding under fine-tuning. Whether midtraining interventions, cleanly isolated from post-training, can produce durable alignment remains untested. We test this via constitutional midtraining: inserting principled, values-based content into midtraining against a replay-only control at 120B scale. Our 394M-token constitutional corpus, built from Anthropic's Constitution, uses a 2x2 factorial design (curriculum ordering x deliberative reasoning) to produce four constitutionally midtrained conditions plus a control, evaluated on self-generated and established benchmarks including alignment under pressure, value conflict resolution, blackmail, and emergent misalignment across three stages: post-midtraining, post-SFT, and post-benign fine-tuning. Constitutionally midtrained models outperform the control on alignment generalization and durability, notably on blackmail: SFT instills a blackmail propensity in all models, but constitutional midtraining blunts it, with the advantage surviving benign fine-tuning (-17.5pp). This durability does not extend to settings requiring active resistance to in-context pressure or conflict, where the advantage attenuates after SFT. The presence of constitutional content at midtraining also matters more than its structure, and constitutional midtraining incurs no cost, on average, on the capabilities we test (MMLU, ARC-Easy, piqa, GSM8K) at any stage. A modest amount of constitutional content at midtraining could therefore yield broad, persistent alignment gains, offering a cheap, complementary addition to SFT-centered pipelines. Code, data, and models are available.

Problem

• alignment durability
• midtraining
• post-training alignment
• constitutional AI
• fine-tuning erosion

Innovation

• Constitutional Midtraining
• Alignment Durability
• Value Conflict Resolution
• Post-training Alignment
• Midtraining Intervention