Less is More: Token Context-aware Learning for Object Tracking

2025-01-01

AI Summary

In visual object tracking, redundant and noisy tokens in the reference frame degrade localization robustness. To address this, we propose LMTrack—a novel framework that introduces reference-token-level importance awareness and autoregressive refinement for the first time. Specifically, we design a Token Context Memory module to model spatiotemporal context, incorporate Unidirectional Token Attention to enforce causal, importance-weighted attention, and integrate dynamic background suppression with importance-driven token sampling. Unlike conventional frame-level coarse-grained modeling, LMTrack performs fine-grained, token-level autoregressive updates. Extensive experiments demonstrate state-of-the-art performance on GOT-10k, TrackingNet, and LaSOT, with significant improvements in long-term tracking accuracy and robustness against occlusion, motion blur, and background clutter.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recently, several studies have shown that utilizing contextual information to perceive target states is crucial for object tracking. They typically capture context by incorporating multiple video frames. However, these naive frame-context methods fail to consider the importance of each patch within a reference frame, making them susceptible to noise and redundant tokens, which deteriorates tracking performance. To address this challenge, we propose a new token context-aware tracking pipeline named LMTrack, designed to automatically learn high-quality reference tokens for efficient visual tracking. Embracing the principle of Less is More, the core idea of LMTrack is to analyze the importance distribution of all reference tokens, where important tokens are collected, continually attended to, and updated. Specifically, a novel Token Context Memory module is designed to dynamically collect high-quality spatio-temporal information of a target in an autoregressive manner, eliminating redundant background tokens from the reference frames. Furthermore, an effective Unidirectional Token Attention mechanism is designed to establish dependencies between reference tokens and search frame, enabling robust cross-frame association and target localization. Extensive experiments demonstrate the superiority of our tracker, achieving state-of-the-art results on tracking benchmarks such as GOT-10K, TrackingNet, and LaSOT.

Problem

• Object Tracking
• Noise Robustness
• Region Importance Assessment

Innovation

• LMTrack
• Unidirectional Token Attention
• Spatio-Temporal Information