Local Support Learning

2026-10-01

AI Summary

This study addresses the problem of catastrophic forgetting during large language model (LLM) fine-tuning by proposing a Local Subspace Learning (LSL) framework. The core innovation lies in formulating forgetting as a geometric problem. Specifically, LSL leverages the likelihood decay properties of Gaussian Mixture Models (GMMs) to construct a local update gating mechanism, which restricts weight adapters to operate exclusively within the current data distribution. This design enables effective knowledge retention without requiring access to historical training data. Experimental results demonstrate that LSL successfully mitigates forgetting in 7-billion-parameter LLMs while maintaining both computational efficiency and scalability.

Abstract

We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.

Problem

• catastrophic forgetting
• large pre-trained models
• continual learning
• knowledge retention

Innovation

• Catastrophic Forgetting
• Local Support Learning
• Gaussian Mixture Model
• Large Language Models
• Parameter-Efficient Fine-Tuning