CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking

2026-07-27

AI Summary

This work addresses the limited robustness of referring expression multi-object tracking (RMOT) methods under cross-domain scenarios and the absence of systematic evaluation of language-guided tracking performance under domain shifts. It formally defines the cross-domain RMOT task and introduces CD-RMOT-Bench, a unified benchmark encompassing real-world data from clear domains, digital twin variants, and videos from adverse domains, enabling multi-dimensional analysis of domain shifts—such as weather and viewpoint—and facilitating synthetic-to-real transfer evaluation. The authors further propose a Query-Centric Adaptation (QCA) framework that achieves robust alignment between linguistic expressions and visual trajectories by stabilizing the query space. Experiments reveal that domain shifts primarily destabilize temporal association, while QCA substantially mitigates performance degradation, establishing a foundation for research on cross-domain robustness in language-guided tracking.

Abstract

Referring multi-object tracking (RMOT) extends tracking from category-driven perception to language-guided understanding by grounding object trajectories in natural-language expressions. Despite recent progress, existing RMOT studies are largely conducted under in-domain settings, leaving the robustness of language-conditioned tracking under inevitable visual domain shifts unexplored. In this paper, we study Cross-Domain Referring Multi-Object Tracking (CD-RMOT), a new and challenging problem that evaluates whether an RMOT model trained on a labeled source domain can reliably follow natural-language expressions in an unlabeled target domain with different visual conditions. To support systematic study, we construct CD-RMOT-Bench, a unified benchmark that combines real clear-domain referring tracking data, aligned digital-twin variants, and real adverse-domain videos. CD-RMOT-Bench enables both controlled weather/viewpoint shift analysis and realistic synthetic-real transfer evaluation under a shared RMOT protocol. Further, we provide a Query-Centric Adaptation (QCA) framework, designed to stabilize the query space that bridges visual trajectories and referring expressions. Extensive experiments reveal that domain shifts severely degrade RMOT performance, where the failure is not merely caused by object detection errors but more critically by unstable expression-conditioned temporal association and target selection. QCA establishes a strong baseline, while CD-RMOT-Bench opens a new direction for robust language-guided tracking across visual domains.

Problem

• Cross-Domain
• Referring Multi-Object Tracking
• Domain Shift
• Language-Guided Tracking
• Visual Domain Adaptation

Innovation

• Cross-Domain Referring Multi-Object Tracking
• CD-RMOT-Bench
• Query-Centric Adaptation
• Language-Guided Tracking
• Domain Shift Robustness