SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning

2026-10-07

AI Summary

This study addresses the catastrophic forgetting of general capabilities in large language models following fine-tuning by proposing a training-free dual-model fusion framework. The method leverages responses from the fine-tuned model as contextual input, guiding the parent model to acquire specialized knowledge through in-context learning. Supported by Bayesian theoretical derivations and attention visualization analyses, this approach achieves complementary enhancement of both specialized and general capabilities, thereby overcoming the performance limitations inherent to single-model paradigms. Experimental results demonstrate that, without additional training, the proposed framework approximates the specialized performance of fine-tuned models across multiple benchmarks while fully preserving the general capabilities of the parent model, significantly outperforming single-model baselines.

Abstract

Supervised fine-tuning (SFT) equips large language models (LLMs) with specialized capabilities, but often comes at the cost of forgetting the general capabilities of their parent models (i.e., the pretrained models before fine-tuning). This trade-off is especially limiting for queries that require both specialized and general capabilities. We introduce SFT-as-context, a training-free method in which the parent model uses the SFT model's response as context to answer the query. This allows the parent model to acquire fine-tuned capabilities from the SFT response through in-context learning while preserving its own general capabilities. Across 19 parent-SFT model pairs and 11 benchmarks, SFT-as-context remains close to the SFT models on fine-tuned capabilities, with gaps of only 2.2 and 2.1 percentage points on AIME 2024 and LiveCodeBench and 2.0 macro MAE on NutriBench-English, while staying within 2.2 percentage points of the parent models on general capabilities on average. Remarkably, it can solve queries requiring both fine-tuned and general capabilities, even when neither the parent nor SFT model succeeds alone. This approach also extends beyond parent-SFT pairs: responses from a small open-source SFT model can improve a strong closed-source LLM, outperforming either model alone. Furthermore, we use a Bayesian framework to derive theoretical guarantees that bound the error of SFT-as-context relative to the SFT model on fine-tuned capabilities and to the parent model on general capabilities. In addition, we visualize the attention weights and find that the parent model attends more to useful SFT responses and less to irrelevant ones, suggesting that selective attention helps the parent model use the SFT response through in-context learning.

Problem

• Supervised Fine-Tuning
• Catastrophic Forgetting
• Large Language Models
• General Capabilities

Innovation

• Supervised Fine-Tuning
• Catastrophic Forgetting
• In-Context Learning
• Training-Free
• Bayesian Framework