Self-Distillation Enables Continual Learning

2026-01-27

AI Summary

This work addresses the challenges of catastrophic forgetting and the lack of effective online policy learning methods in continual learning for foundation models. To overcome these limitations, the authors propose Self-Distillation Fine-Tuning (SDFT), a novel approach that enables online policy self-distillation using only expert demonstrations. SDFT leverages in-context learning to generate training data by treating the model’s own outputs—conditioned on provided demonstrations—as teacher signals, thereby eliminating the need for explicit reward functions. This mechanism simultaneously preserves previously acquired knowledge while acquiring new skills. Experimental results demonstrate that SDFT substantially outperforms conventional supervised fine-tuning, achieving superior performance on new tasks, effectively mitigating catastrophic forgetting, and enabling the sequential accumulation of multiple skills without performance degradation.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations.

Problem

• continual learning
• catastrophic forgetting
• expert demonstrations
• foundation models
• on-policy learning

Innovation

• Self-Distillation
• Continual Learning
• On-Policy Learning
• Demonstration-Based Learning
• Catastrophic Forgetting