RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting

2026-05-28

AI Summary

This work addresses the degradation of general-purpose capabilities in domain-specific supervised fine-tuning, often caused by supervision incompatibility and trajectory preservation gaps. The authors propose a two-stage framework: first, they construct model-compatible supervision data through self-conditioned rewriting, semantic filtering, and answer fusion; second, they introduce answer-conditioned online policy distillation, integrating Top-K temperature distillation with EMA-based adaptive loss balancing to preserve trajectory-level behavior. This approach uniquely unifies data refinement and behavioral alignment mechanisms, achieving an average 23.2% improvement in domain accuracy across three base models and five domains, while recovering 18.2% and 10.2% of lost general-purpose performance on MS-Bench and IFEval, respectively.

Abstract

Domain-specific supervised fine-tuning (SFT) often improves in-domain performance at the cost of degrading a model's general capabilities. We view this degradation through two practical gaps in domain SFT: a supervision-compatibility gap, where domain targets differ in style and reasoning format from the original model's natural responses, and a trajectory-preservation gap, where teacher-forced SFT optimizes fixed target tokens without constraining the model's behavior on its own generated prefixes. This process fails to preserve the model's original behavior. We propose RAFT (Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting), a two-stage framework that addresses both factors. First, RAFT constructs model-compatible supervision through self-conditioned rewriting, semantic filtering, and answer fusion. Second, RAFT performs Answer-Conditioned On-Policy Distillation, where the original instruction-tuned model provides soft targets on student-generated trajectories while being conditioned on the fused answer as helpful context. We further introduce top-K temperature distillation and EMA-based adaptive loss balancing to stabilize the domain-general trade-off. Across three instruction-tuned backbones and five domains, RAFT improves average domain accuracy by 23.2% over standard SFT, while recovering part of the SFT-induced degradation on MS-Bench and IFEval, with relative improvements of 18.2% and 10.2%, respectively. These results show that coupling data refinement with trajectory-level preservation provides an effective recipe for domain fine-tuning with alleviated forgetting.

Problem

• catastrophic forgetting
• supervised fine-tuning
• domain adaptation
• trajectory preservation
• supervision compatibility

Innovation

• Data Refinement
• Adaptive Distillation
• On-Policy Distillation
• Alleviated Forgetting
• Domain Fine-Tuning