linear readout training

This work addresses the stability-plasticity trade-off in continual learning for large language models, particularly highlighting how experience replay can induce negative transfer and performance collapse in structured tasks such as code generation. The study is the first to reveal a divergence in the efficacy of experience replay between structured and unstructured tasks. To mitigate this issue, the authors propose Orthogonal Subspace Wake-up (OSW), a mechanism that identifies task-relevant parameter subspaces, retrieves critical parameters during a “wake-up” phase, and applies orthogonal gradient updates to ensure new learning does not interfere with previously acquired knowledge structures. Evaluated on a sequence of four heterogeneous tasks including code generation, OSW effectively prevents the degradation of coding capabilities caused by experience replay while maintaining strong plasticity for efficient new task acquisition, thereby achieving a synergistic balance between structural safety and adaptability.

Balancing model stability and plasticity remains a fundamental challenge in continual learning. This paper proposes Flashback Learning (FL), the first bidirectional regularization framework that decouples and jointly optimizes these competing objectives. FL introduces two complementary knowledge repositories— an *old knowledge base* to enhance stability and a *new knowledge base* to boost plasticity—driving a two-stage training process. It supports plug-and-play integration with mainstream approaches, including experience replay, knowledge distillation, parameter regularization, and dynamic architectures. Theoretical analysis elucidates FL’s mechanism for achieving stability–plasticity equilibrium. Empirically, FL achieves average accuracy improvements of +4.91% under class-incremental and +3.51% under task-incremental settings, significantly enhancing the stability–plasticity trade-off ratio. Moreover, it outperforms state-of-the-art methods on challenging benchmarks such as ImageNet.

This work addresses the challenges of knowledge compression and inter-task interference in continual learning, which are exacerbated by energy dispersion across model parameters. To mitigate these issues, the authors propose Energy-concentrated and Ordered Low-Rank Adaptation (E²-LoRA), a novel mechanism that analyzes the low-rank structure of output feature drift. By preserving critical parameters along principal component directions to minimize reconstruction error, E²-LoRA explicitly compresses and orders accumulated knowledge into leading singular subspaces, thereby freeing model capacity for subsequent tasks. Coupled with a dynamic rank allocation strategy, the method jointly optimizes knowledge retention and model plasticity. Extensive experiments on multiple continual learning benchmarks demonstrate that E²-LoRA significantly outperforms existing approaches, achieving state-of-the-art performance.

Existing replay-based continual learning methods suffer from poor scalability and high computational overhead, while failing to effectively integrate parameter-efficient fine-tuning techniques—such as Low-Rank Adaptation (LoRA) and model merging—into multi-task adaptation. To address this, we propose a novel replay framework that deeply couples staged knowledge consolidation with sequential model merging tailored for task-ordered streams. It is the first to systematically integrate LoRA, task arithmetic, and collaborative optimization within a unified replay paradigm. Our approach reduces required replay sample volume by up to 55%, significantly improving training efficiency and model scalability. Empirically, it outperforms individual baseline methods across diverse multi-task sequences, achieving organic synergy between replay mechanisms and parameter-efficient adaptation. This work establishes a new paradigm for lightweight, accumulative continual learning models.

Existing MR reconstruction methods prioritize image fidelity while neglecting their impact on downstream tasks (e.g., segmentation, classification), leading to cascaded performance degradation due to error propagation and domain shift. To address this, we propose a continual learning–based reconstruction optimization framework tailored for sequential multi-task deployment. For the first time, we introduce replay-based continual learning into MR reconstruction fine-tuning: a replay buffer jointly optimizes k-space domain reconstruction and downstream task gradients via a multi-task weighted loss, effectively mitigating catastrophic forgetting. Our approach employs a single reconstruction network that concurrently adapts to multiple downstream tasks—preserving high performance across all tasks while eliminating cascade mismatch. Extensive experiments demonstrate that our method significantly outperforms the conventional dual-network paradigm relying on independent optimization.

This work addresses a critical gap in continual learning research: while most existing methods focus on mitigating catastrophic forgetting, they largely overlook the conditions under which forward transfer—where knowledge from past tasks benefits new ones—can be effectively realized. The paper introduces, for the first time, a systematic three-condition framework that characterizes when forward transfer is feasible and proposes Transfer-Selective Replay (TSR), a novel method that leverages a zero-training-overhead task signature mechanism to automatically identify and replay only those historical samples beneficial to the current task. TSR integrates knowledge distillation to preserve performance on previous tasks while explicitly promoting forward transfer as a first-class objective. Experiments demonstrate that TSR significantly enhances forward transfer across both homogeneous and heterogeneous task sequences and consistently outperforms existing replay-based baselines, especially under limited replay budgets.

This work addresses the challenge of catastrophic forgetting in continual learning, where parameter updates for new tasks often interfere with previously acquired knowledge, and conventional regularization methods inefficiently utilize model capacity. The authors propose a proximal decoupling framework that reformulates the stability-plasticity trade-off as the coordinated update of two complementary operators: first optimizing the loss for the current task, then applying proximal sparse regularization to preserve critical parameters while pruning redundant ones. Built upon operator splitting and proximal gradient algorithms, this approach avoids gradient conflicts without relying on replay buffers, Bayesian sampling, or meta-learning components. Evaluated on standard continual learning benchmarks, the method demonstrates superior long-term stability and adaptability across extended task sequences, achieving state-of-the-art performance.

This work addresses catastrophic forgetting in continual learning under non-stationary data streams by proposing the COLD framework, which introduces, for the first time, the Drift-Plus-Penalty stochastic optimization method from control theory into this domain. COLD formulates forgetting as a controlled dynamic process, employing virtual queues to track performance deviations on historical tasks and jointly minimizing the current task loss and queue drift at each optimization step. This mechanism explicitly governs the stability-plasticity trade-off. The framework provides theoretical guarantees on stability and convergence, and achieves significantly superior performance over state-of-the-art methods on standard benchmarks, enabling controllable and efficient suppression of catastrophic forgetting.

This work addresses the tendency of mLSTM models to become trapped in prolonged training plateaus on noisy associative recall tasks. To mitigate this issue, the authors propose imposing orthogonality constraints—via Newton–Schulz iteration or recursive least squares—on the memory matrix during the readout phase, thereby improving the condition number of the learning problem and accelerating optimization. This mechanism functions as a removable training scaffold: it does not increase the model’s intrinsic memory capacity, yet boosts the probability of escaping plateaus by approximately sixfold and substantially widens the effective learning rate range. After removal, the model retains full recall accuracy, and in failure cases, nearly half of the associated information remains recoverable through linear decoding. The findings suggest that current recall performance reflects trainability more than architectural capacity, offering an interpretable mechanism underlying apparent “emergent” behavior.

This work addresses catastrophic forgetting in continual learning by introducing a geometric perspective that frames forgetting as interference between tasks, quantified as an interference energy. Under frozen feature representations, the authors precisely measure this interference using the path-averaged curvature and theoretically link task support set separability to parameter retainability. Building on this insight, they propose a task-aware orthogonalization mechanism that enables structured parameter isolation and sharing, adaptively allocating parameters without requiring rehearsal or Fisher information. The method achieves lossless retention on structurally separable tasks and transforms irreversible forgetting into recoverable plasticity on non-separable tasks, consistently outperforming existing replay-free approaches across both similar and dissimilar task sequences.