PHASER: Phase-Aware and Semantic Experience Replay for Vision-Language-Action Models

2026-06-02

AI Summary

This work addresses stage starvation and task forgetting in vision-language-action models during continual learning, which arise from imbalanced experience replay. To mitigate these issues, the authors propose an architecture-agnostic continual learning framework that incorporates a stage-aware memory allocation mechanism and a multimodal interference routing strategy. This approach dynamically prioritizes the replay of critical sub-skill stages prone to forgetting, leveraging unsupervised action change-point detection and semantic validation from vision-language models to automatically identify and preserve semantically coherent manipulation phases. Evaluated on the LIBERO continual learning benchmark, the method achieves a 31% average improvement in success rate under identical replay budgets and attains a final success rate of 87.8% in the LIBERO-Goal CL setting, marking the first demonstration of balanced lifelong retention of key sub-skills within manipulation trajectories.

Abstract

Vision-Language-Action (VLA) models have achieved remarkable success in language-conditioned robotic manipulation. However, deploying these models in open-ended environments requires continuously acquiring novel skills, a process that inevitably triggers severe catastrophic forgetting of previously learned behaviors. While experience replay (ER) serves as a standard mitigating strategy, naive uniform sampling fundamentally misaligns with the temporal characteristics of manipulation trajectories. It systematically under-samples brief but causally critical sub-skills, leading to phase starvation, and completely overlooks the varying degrees of forgetting across historical tasks. To overcome these limitations, we introduce PHASER, an architecture-agnostic continual learning framework. PHASER employs a phase-centric capacity allocation to guarantee equal memory support for all sub-skills, coupled with a multi-modal interference routing strategy that dynamically prioritizes historical phases at high risk of forgetting. Furthermore, to enable fully autonomous lifelong adaptation, we integrate Auto-PC, a lightweight pipeline combining unsupervised action-signal change-point detection with VLM-based semantic verification to extract temporal boundaries without intensive manual supervision. Evaluated across three VLA backbones on LIBERO continual learning suites, PHASER yields substantial empirical improvements, increasing Average Success Rate (ASR) by up to 31% over matched-budget ER and achieving an 87.8% final ASR on the LIBERO-Goal CL setting.

Problem

• catastrophic forgetting
• experience replay
• vision-language-action models
• continual learning
• phase starvation

Innovation

• experience replay
• phase-aware learning
• catastrophic forgetting
• vision-language-action models
• continual learning