Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models

2026-09-26

AI Summary

This study addresses the challenges of historical prompt invalidation and the lack of explicit cross-task relationship modeling in the continual adaptation of large language models. To this end, we propose LAPS, a method that pioneers an anchored prompt space utilizing learnable Bézier control points to explicitly characterize inter-task geometric relationships, thereby facilitating positive transfer. Furthermore, LAPS introduces a self-distillation mechanism to align historical prompts, combined with soft prompt tuning and joint parameter optimization to enhance adaptation efficiency. Experimental results demonstrate that LAPS significantly outperforms existing baselines on the TRACE benchmark, effectively improving average performance while mitigating catastrophic forgetting.

Abstract

Continually adapting large language models requires acquiring new knowledge while preserving previously learned capabilities. Jointly adapting model parameters and task-specific soft prompts offers a promising solution, but faces two key limitations: historical prompts may become less effective as the model evolves, while their transferable cross-task relationships are not explicitly learned. We propose Learning an Anchored Prompt Space (LAPS), which preserves historical prompt effectiveness and learns relationships among task-specific soft prompts to facilitate positive transfer. LAPS first aligns historical prompts with the updated model through self-distillation. LAPS then constructs an anchored prompt space whose vertices correspond to learned task-specific soft prompts and whose intermediate geometry is shaped by learnable B\'ezier control prompts. Once this anchored prompt space is learned, LAPS identifies the best-performing prompt for each task on its validation set, allowing the optimized prompt to draw on knowledge acquired from observed tasks. Experiments on the TRACE benchmark across three Qwen3 model scales show that LAPS consistently outperforms distillation-based, prompt-based, and joint prompt--parameter adaptation baselines, improving average performance while reducing forgetting.

Problem

• Continual Adaptation
• Large Language Models
• Soft Prompts
• Catastrophic Forgetting
• Prompt Space

Innovation

• Continual Adaptation
• Anchored Prompt Space
• Self-Distillation
• Bézier Control Prompts
• Soft Prompts