Problem
Research questions and friction points this paper is trying to address.
catastrophic forgetting
fine-tuning
foundation models
Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification
🤖 AI Summary
本文提出了一种基于JANUS修正的后处理框架，通过将参数更新投影到Jacobian Null Space来解决微调基础模型时的历史知识遗忘问题。
📝 Abstract
Fine-tuning foundation models on new tasks inevitably suffer from catastrophic forgetting. While existing works attempt to mitigate this on the basis of parameter-efficient fine-tuning methods, they adopted an overly restrictive Subspace Orthogonality condition. In this paper, we introduce a purely post-hoc and tuning-agnostic weight rectification framework that achieves Parameter Space Orthogonality, which is the necessary and sufficient condition for preserving historical performance to the first order. By projecting parameter updates into the JAcobian NUll Space (JANUS), our method significantly recovers compromised historical knowledge without interfering with the underlying fine-tuning process. To overcome the local validity of the Jacobian approximation, we further propose a Multi-step Adaptive Rectification mechanism that utilizes the JANUS shift to dynamically verify the valid trust region and adjust step sizes. Coupled with our proposed ghost projection, ghost orientation comparison, and sequence-level singular value decomposition compression techniques, JANUS also achieves great temporal and spatial efficiency. Experiments demonstrate that JANUS seamlessly integrates with various fine-tuning methods, significantly mitigating the stability-plasticity dilemma by recovering historical knowledge while preserving downstream task adaptation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Post-hoc Weight Rectification
Parameter Space Orthogonality
Multi-step Adaptive Rectification
Jacobian Null Space
🔎 Similar Papers
No similar papers found.
