RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detection

2025-08-19

AI Summary

Existing incremental object detection (IOD) benchmarks rely heavily on synthetic or simplified datasets, failing to reflect real-world challenges in adapting to novel classes or domains while preserving previously learned knowledge. Method: We introduce two realistic IOD benchmarks—D-RICO and EC-RICO—spanning 14 diverse real-world datasets, explicitly modeling both domain shift and class-incremental learning. We propose the first unified evaluation framework integrating real and synthetic data, multi-domain dynamics, and annotation heterogeneity, with distinct protocols for domain-incremental (DI) and class-incremental (CI) settings. Contribution/Results: Extensive experiments reveal that state-of-the-art IOD methods underperform even simple replay baselines; independent per-task training remains optimal. Our analysis uncovers fundamental limitations in current approaches—including inadequate teacher guidance, poor task unification, and insufficient plasticity. This work establishes more credible evaluation standards and identifies critical research directions for realistic incremental object detection.

Technology Category

• Computer Vision: Object Detection & Categorization
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Incremental Learning (IL) trains models sequentially on new data without full retraining, offering privacy, efficiency, and scalability. IL must balance adaptability to new data with retention of old knowledge. However, evaluations often rely on synthetic, simplified benchmarks, obscuring real-world IL performance. To address this, we introduce two Realistic Incremental Object Detection Benchmarks (RICO): Domain RICO (D-RICO) features domain shifts with a fixed class set, and Expanding-Classes RICO (EC-RICO) integrates new domains and classes per IL step. Built from 14 diverse datasets covering real and synthetic domains, varying conditions (e.g., weather, time of day), camera sensors, perspectives, and labeling policies, both benchmarks capture challenges absent in existing evaluations. Our experiments show that all IL methods underperform in adaptability and retention, while replaying a small amount of previous data already outperforms all methods. However, individual training on the data remains superior. We heuristically attribute this gap to weak teachers in distillation, single models' inability to manage diverse tasks, and insufficient plasticity. Our code will be made publicly available.

Problem

• Evaluates incremental learning in object detection with realistic benchmarks
• Addresses performance gaps in adaptability and knowledge retention
• Identifies limitations of synthetic benchmarks in real-world scenarios

Innovation

• Realistic benchmarks with domain shifts
• Replaying previous data improves performance
• Identifies distillation and plasticity limitations