PILOT: A Data-Free Continual Learning Approach for Real-Time Semantic Segmentation via Boundary Guidance

2026-05-26

AI Summary

This work addresses catastrophic forgetting in real-time semantic segmentation models during incremental learning when new classes are introduced. To tackle this challenge, the authors propose PILOT, a lightweight continual learning framework tailored for PIDNet. PILOT freezes the original network parameters and introduces a parallel derivative branch (D-branch) to capture high-frequency boundary information of new classes, enabling knowledge increment without access to historical data. By leveraging a boundary-guided mechanism and a data-agnostic learning strategy, PILOT effectively mitigates forgetting and substantially reduces training overhead while incurring negligible inference latency. Experimental results demonstrate that PILOT achieves accurate segmentation on newly added classes while preserving high mIoU for original classes, outperforming current state-of-the-art continual learning approaches.

Abstract

Real-time semantic segmentation models offer an excellent balance between accuracy and inference speed. However, deploying these models in dynamic real world environments often requires the ability to learn novel classes incrementally without retraining on the entire dataset. This capability is known as continual learning. In this regard, the standard fine-tuning methods in deep learning often fail due to catastrophic forgetting, where the model learns new information but forgets previously trained and learned classes. Contributing to this crucial domain, the current paper proposes a novel continual learning framework tailored for PIDNet, which is a widely cited state-of-the-art real-time semantic segmentation model. Our method, PILOT(Parallel Incremental Learning Over Time), introduces a real-time and lightweight strategy by implementing a parallel Derivative-branch (D-branch) designed to capture the high frequency boundary information of novel classes while freezing the trained parameters of the original segmentation network. This novel setup allows the model to adapt to new semantic categories while preserving the knowledge of previously learned classes. By using only data associated with the new class, our model significantly reduces training overhead. Experimental results demonstrate that our approach successfully segments new classes while maintaining high mean Intersection over Union (mIoU) on the original base classes, thereby comfortably outperforming all major continual learning approaches in this domain. Overall, PILOT is shown to effectively mitigate catastrophic forgetting with minimal impact on inference latency, thus maintaining real-time performance.

Problem

• continual learning
• semantic segmentation
• catastrophic forgetting
• real-time
• incremental learning

Innovation

• continual learning
• real-time semantic segmentation
• catastrophic forgetting
• boundary guidance
• data-free