Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models

2026-09-25

AI Summary

This study addresses the challenge of catastrophic forgetting and general knowledge degradation in the continual fine-tuning of large language models (LLMs), which arises from the inaccessibility of pre-training gradients. To tackle this issue, we propose the EoupCT framework, which pioneers the use of learnable soft prompts to generate pseudo-data for approximating unknown pre-training gradients. By incorporating Gumbel-Softmax relaxation techniques, the framework introduces an efficient first-order Pareto optimizer that performs orthogonal gradient projection to synergistically optimize multi-objective tasks. Extensive experiments across multiple LLMs demonstrate that EoupCT effectively mitigates catastrophic forgetting, achieving a significant balance between downstream task performance and the preservation of the model’s inherent general knowledge.

Abstract

Continual fine-tuning is essential for large language models (LLMs) to dynamically adapt to real-world environments, yet it inevitably suffers from catastrophic forgetting, particularly the performance degradation of previous tasks and LLMs' general-purpose knowledge. Although existing methods, such as orthogonal gradient projection, mitigate the forgetting across various fine-tuning tasks, they fundamentally fail to preserve pre-training LLMs' inherent general-purpose knowledge because the original data and gradients of off-the-shelf pre-training LLMs required by these methods are strictly unknown and highly diverse. To bridge this critical gap, we propose EoupCT, a novel framework designed to Estimate and Orthogonalize Unknown Pre-training gradients for Continual LLM fine-Tuning. Specifically, EoupCT estimates pre-training gradients by dynamically generating pseudo data that is most susceptible to forgetting for new tasks through a learnable soft prompt equipped with Gumbel-Softmax relaxation. Furthermore, we formulate a multi-objective optimization problem and introduce a first-order efficient Pareto optimizer that jointly optimizes LLM parameters and the soft prompt, rigorously enforcing orthogonality between new task updates and the estimated pre-training gradients. Extensive experiments across multiple LLMs demonstrate that EoupCT effectively preserves both task-specific proficiency and inherent general-purpose knowledge, successfully mitigating the catastrophic forgetting.

Problem

• Continual Fine-tuning
• Catastrophic Forgetting
• Large Language Models
• Unknown Pre-training Gradients
• General-purpose Knowledge

Innovation

• Continual Fine-tuning
• Catastrophic Forgetting
• Gradient Orthogonalization
• Soft Prompt
• Multi-objective Optimization