CL-CLIP: CLIP-Based Continual Learning Framework with Cost-Volume Category Decoupling for Object Detection

2026-06-05

AI Summary

This work addresses catastrophic forgetting in continual object detection, where models tend to forget previously learned categories when acquiring new ones. To mitigate this issue, the authors propose a CLIP-based continual learning framework that incorporates a cost-volume-guided category decoupling mechanism. By leveraging CLIP’s image-text similarity to construct a cost volume, the method decomposes region features into category-specific pathways and introduces a multi-expert RoI detection head to jointly optimize learning of new classes and retention of old ones. The approach effectively integrates CLIP’s zero-shot priors into the continual learning process, substantially alleviating forgetting. Experimental results on PASCAL VOC and MS-COCO demonstrate that the proposed method outperforms existing baselines such as F-ViT, achieving state-of-the-art performance in both adapting to novel categories and preserving accuracy on base categories.

Abstract

Continual Object Detection (COD) requires a detector to acquire new categories over time while preserving previously learned ones. This goal is closely related to open-vocabulary detection, since both settings require reasoning over categories that are not fully covered by the annotations available at the current training stage. Recent CLIP-based open-vocabulary detectors have shown strong zero-shot generalization, and frameworks such as F-ViT demonstrate that vision-language pretraining can provide powerful zero-shot detection ability for unseen categories. However, real-world deployments cannot remain purely zero-shot: once these detectors are continually updated on newly introduced categories, they suffer severe catastrophic forgetting and quickly lose their previously calibrated detection ability. We therefore propose CL-CLIP, a CLIP-based COD framework that equips open-vocabulary detectors with better continual learning ability through cost-volume-guided category decoupling. Specifically, following CAT-Seg, we compute a CLIP image-text similarity cost volume, defined as dense category-wise response maps between visual tokens and class text embeddings. This zero-shot spatial prior decomposes shared region features into class-specific pathways, which are then processed by a Multi-Expert RoI head. Extensive experiments on PASCAL VOC and MS-COCO show that CL-CLIP substantially improves the F-ViT baseline under continual fine-tuning and achieves competitive performance with existing continual object detectors, especially in adapting to newly introduced categories while preserving competitive base-class performance.

Problem

• Continual Object Detection
• Catastrophic Forgetting
• Open-Vocabulary Detection
• CLIP
• Zero-Shot Generalization

Innovation

• Continual Learning
• CLIP
• Cost Volume
• Category Decoupling
• Open-Vocabulary Detection