STAIL: Semantic Text-Anchored Incremental Learning for Medical Imaging via Large Language Models

2026-08-06

AI Summary

This work addresses catastrophic forgetting, high storage overhead, and privacy risks in continual learning for medical imaging caused by pixel-level replay. To overcome these challenges, the authors propose Semantic Text-Anchored Incremental Learning (STAIL), a novel framework that leverages a small set of image anchors paired with rich textual descriptions to reconstruct knowledge of previous tasks within the frozen semantic space of a large language model. STAIL introduces an asymmetric semantic consolidation buffer and a large language model–based semantic anchoring mechanism, using text as a stable prior to guide the evolution of visual features. Evaluated on retinal, ultrasound, and X-ray medical images, STAIL significantly outperforms baseline methods, achieving average improvements of 2.24% in AAA-AUC and 3.55% in BWT-AUC, thereby effectively mitigating forgetting while enabling efficient, low-storage, and privacy-preserving continual learning.

Abstract

Deep learning models applied to medical image analysis suffer from severe catastrophic forgetting when continually adapting to new clinical tasks in dynamic environments. Mainstream incremental learning methods typically mitigate this by rehearsing raw historical images. However, this pixel-level rehearsal incurs significant storage overhead, raises privacy concerns, and fails to adequately capture the true data distribution with sparse exemplars. Inspired by human cognitive mechanisms, we propose a novel framework termed Semantic Text-Anchored Incremental Learning (STAIL) for sequential clinical tasks. To overcome the rehearsal bottleneck, STAIL introduces an asymmetric semantic consolidation buffer (SCB). By incorporating a minimal set of image anchors and extensive textual descriptions, the SCB enables dense semantic reconstruction of old tasks at a minimal storage cost. Furthermore, we design an LLM-derived Semantic Anchoring Mechanism (LSAM) that leverages the stable semantic space of frozen large language models as developmental priors. This mechanism explicitly anchors evolving visual features to textual representations, guiding and constraining plasticity and stability at both macroscopic and microscopic levels. Extensive experiments across three heterogeneous medical datasets, covering fundus, ultrasound, and X-ray imaging, demonstrate that STAIL acts as a highly effective plug-and-play module. It comprehensively enhances the performance of various existing baselines, achieving average gains of 2.24\% in AAA-AUC for sustained performance and 3.55\% in BWT-AUC for reduced forgetting. Code is available.

Problem

• catastrophic forgetting
• incremental learning
• medical imaging
• rehearsal
• data distribution

Innovation

• incremental learning
• semantic anchoring
• large language models
• medical imaging
• catastrophic forgetting