OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback

2025-11-01

AI Summary

Panoramic multi-object tracking (MOT) faces fundamental challenges including geometric distortion, resolution degradation, and identity ambiguity induced by the 360° field of view. To address these, this paper proposes FlexiTrack—a feedback-driven, end-to-end tracking framework. FlexiTrack introduces a novel trajectory feedback mechanism and dynamic state-space modeling (DynamicSSM) to enable robust long-term identity preservation via an expert mixture memory module (ExpertTrack). Additionally, a tracklet management module adaptively selects optimal tracking strategies based on scene dynamics. Evaluated on the JRDB and our newly established EmboTrack benchmarks, FlexiTrack achieves absolute improvements of 25.5% and 43.07% in HOTA, respectively, significantly outperforming existing methods. The framework establishes a new paradigm for panoramic MOT that is robust, accurate, and scalable.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Security and Privacy: Tracking, profiling, and countermeasures against them
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper investigates Multi-Object Tracking (MOT) in panoramic imagery, which introduces unique challenges including a 360° Field of View (FoV), resolution dilution, and severe view-dependent distortions. Conventional MOT methods designed for narrow-FoV pinhole cameras generalize unsatisfactorily under these conditions. To address panoramic distortion, large search space, and identity ambiguity under a 360° FoV, OmniTrack++ adopts a feedback-driven framework that progressively refines perception with trajectory cues. A DynamicSSM block first stabilizes panoramic features, implicitly alleviating geometric distortion. On top of normalized representations, FlexiTrack Instances use trajectory-informed feedback for flexible localization and reliable short-term association. To ensure long-term robustness, an ExpertTrack Memory consolidates appearance cues via a Mixture-of-Experts design, enabling recovery from fragmented tracks and reducing identity drift. Finally, a Tracklet Management module adaptively switches between end-to-end and tracking-by-detection modes according to scene dynamics, offering a balanced and scalable solution for panoramic MOT. To support rigorous evaluation, we establish the EmboTrack benchmark, a comprehensive dataset for panoramic MOT that includes QuadTrack, captured with a quadruped robot, and BipTrack, collected with a bipedal wheel-legged robot. Together, these datasets span wide-angle environments and diverse motion patterns, providing a challenging testbed for real-world panoramic perception. Extensive experiments on JRDB and EmboTrack demonstrate that OmniTrack++ achieves state-of-the-art performance, yielding substantial HOTA improvements of +25.5% on JRDB and +43.07% on QuadTrack over the original OmniTrack. Datasets and code will be made publicly available at https://github.com/xifen523/OmniTrack.

Problem

• Tracking multiple objects in panoramic imagery with 360° field of view
• Addressing severe distortions and identity ambiguity in omnidirectional vision
• Improving long-term robustness against fragmented tracks and identity drift

Innovation

• Feedback-driven framework refines perception with trajectory cues
• DynamicSSM block stabilizes panoramic features and alleviates distortion
• ExpertTrack Memory consolidates appearance cues via Mixture-of-Experts design