Continual Learning without Continual Training

2026-10-07

AI Summary

This study addresses the limitations of conventional continual learning, which relies on gradient updates and is prone to catastrophic forgetting while struggling to accommodate both class- and domain-incremental settings. We propose a frozen model based on Prior Fitted Networks (PFNs) that replaces continual training with continual inference. By performing contextual Bayesian inference within a latent concept space and dynamically expanding the evidence set, our method achieves inference-based adaptation without parameter updates, effectively tackling task-free incremental learning. Experimental results demonstrate that this framework attains competitive performance on both class- and domain-incremental benchmarks, significantly mitigates forgetting rates, and enables interpretable latent concept learning.

Abstract

Continual learning requires models to adapt to new domains and new classes while retaining prior knowledge. Many existing methods rely on continued optimization, using regularization, replay, or parameter expansion to prevent new updates from overwriting previously learned knowledge. Instead, we propose replacing continual training with continual inference: a PFN-based model that is meta-trained, and then frozen, adapting to new classes only by extending an in-context evidence set. Our model, Latent Concept PFN, performs in-context Bayesian inference over a latent concept space that captures semantic structure shared across domains and classes. As each new domain or class arrives, exemplars are added to the memory; adaptation reflects updated posterior beliefs over latent concepts rather than gradient updates. No parameters are changed, reducing forgetting. The same method handles both domain and class incremental continual learning without task identity. Concept annotations are only used during meta-training, acting as a soft anchor on the latent space rather than a fixed bottleneck. Unlike fixed-vocabulary concept methods, the model also handles noisy, ambiguous, or incomplete annotations by combining concept labels with raw input evidence to discover distinctions beyond the predefined concept set. Experiments on class and domain incremental learning datasets demonstrate competitive continual learning performance while learning interpretable latent concepts.

Problem

• Continual Learning
• Catastrophic Forgetting
• Class Incremental Learning
• Domain Incremental Learning
• In-context Learning

Innovation

• Continual Inference
• Prior-Data Fitted Networks (PFN)
• In-Context Bayesian Inference
• Latent Concept Space
• Parameter-Free Adaptation