AnyTrack: Unifying Visual Object Tracking with Any Modalities

2026-08-06

AI Summary

Existing visual object tracking methods struggle to generalize across arbitrary modality combinations and suffer significant performance degradation when modalities are missing. To address this limitation, this work proposes AnyTrack, the first unified tracking framework capable of handling any input modality—including images, language, and audio—through a modality-aware interaction module (MIM) that enables dynamic cross-modal fusion. Additionally, a context understanding module (CUM) is introduced to model contextual information by integrating global and local cues. Evaluated on a newly constructed multimodal tracking benchmark, AnyTrack achieves state-of-the-art performance under both complete and missing modality settings, substantially enhancing model flexibility and robustness.

Abstract

Visual object tracking aims to continuously locate specific targets within sequential frames, evolving from single-modal methods to multi-modal ones. However, existing multi-modal trackers are typically designed for fixed modality combinations, requiring separate models for different inputs. This leads to a poor adaptability to missing or imperfect modalities, and limited generalization. To address these issues, we propose a novel unified framework called AnyTrack for object tracking with any modalities. Specifically, we design a Modality-aware Interaction Module (MIM) to facilitate dynamic interaction across diverse modalities. This module bridges modality discrepancies and aggregates temporal cues to maintain spatio-temporal consistency during cross-modal interaction. Furthermore, we introduce a Context Understanding Module (CUM) to establish spatial correspondence between visual features and target locations via global-local prompts. This module employs target-aware context modeling to enhance foreground-background discrimination for precise localization. Finally, to support the training and evaluation under diverse modalities, we extend existing multi-modal object tracking benchmarks by incorporating grayscale images, language descriptions, and audio clips. Extensive experiments with both complete and missing modality settings demonstrate that our AnyTrack achieves state-of-the-art performance, validating its effectiveness and flexibility. The source code is available at https://github.com/IdolLab/AnyTrack.

Problem

• visual object tracking
• multi-modal
• modality adaptability
• generalization
• missing modalities

Innovation

• unified multi-modal tracking
• modality-aware interaction
• context understanding module
• spatio-temporal consistency
• any-modality adaptation