tracking integration

Video Object Segmentation and Tracking (VOST) suffers from poor temporal consistency, limited generalization, and low computational efficiency. To address these challenges, this paper presents a systematic survey of SAM- and SAM2-based VOST methods and proposes a foundation-model-driven paradigm: (1) a motion-aware memory selection mechanism to mitigate error accumulation; (2) trajectory-guided prompting to enhance temporal robustness; and (3) integration of streaming memory architecture, dynamic feature extraction, and motion prediction for efficient inference. Experiments demonstrate that the framework achieves superior trade-offs between accuracy and real-time performance. Furthermore, the study identifies critical bottlenecks—including memory redundancy, suboptimal prompt efficiency, and long-term error propagation—offering, for the first time, a structured technical roadmap and concrete future research directions for adapting SAM to VOST.

General object tracking remains challenging due to occlusion, appearance variation, and similar distractors. This paper systematically surveys three mainstream paradigms—Siamese, discriminative, and Transformer-based approaches—with particular emphasis on the emerging Transformer-based paradigm. We propose a novel taxonomy tailored for general-purpose trackers and establish a unified, visualization-enhanced, structured comparison framework, analyzing methods across model architecture, spatiotemporal modeling mechanisms, and training strategies. Quantitative evaluation is conducted on major benchmarks including LaSOT and TrackingNet. Results demonstrate that Transformer-based trackers achieve superior robustness and generalization, primarily owing to their global contextual modeling and long-range dependency capture capabilities. The survey clarifies the technical evolution trajectory of general tracking, identifies current bottlenecks—including computational overhead, data hunger, and domain adaptability—and highlights promising future directions such as lightweight design, few-shot adaptation, and cross-domain transfer learning.

The performance bottlenecks among classification, segmentation, and tracking in video instance segmentation remain unclear, hindering targeted improvements. This work proposes the first systematic error decoupling framework, formulating identity and category assignment as an integer linear programming (ILP) problem to construct a model-agnostic diagnostic oracle, accompanied by a visualization tool named TrackLens. Analyses of seven methods on YouTube-VIS and OVIS reveal that tracking instability is the primary bottleneck, stemming from algorithmic temporal fragility rather than representational capacity. Online methods suffer a tracking AP drop exceeding 20% under severe occlusion, with performance degrading sharply as video length and instance density increase—an issue not substantially mitigated by stronger backbone networks.

Existing point tracking methods face significant challenges in real-world scenarios—including high motion complexity, frequent occlusions, and large object diversity—yet lack a systematic benchmark for evaluating robustness and failure modes. To address this, we introduce ITTO, the first high-challenge dynamic point tracking benchmark, comprising first-person real-world videos and multi-source data. We propose a multi-stage manual annotation protocol to precisely characterize motion patterns, occlusion events, and appearance variations. ITTO introduces a novel performance analysis protocol stratified by motion complexity and, for the first time, systematically exposes critical failure points of mainstream trackers—particularly in post-occlusion re-identification. Experiments reveal substantial performance degradation of state-of-the-art methods on ITTO, highlighting deficiencies in long-term occlusion handling and complex dynamic modeling. These findings provide concrete diagnostic insights and quantitative evaluation standards to guide algorithmic improvement.

Existing video understanding methods often rely on modality-specific architectures or parameters, hindering simultaneous cross-modal generalization and multi-task synergy while neglecting modality distribution shifts and task representation heterogeneity. To address this, we propose the first universal framework for video object tracking and segmentation applicable to arbitrary modalities. Our approach introduces a decoupled Mixture-of-Experts (DeMoE) mechanism that explicitly separates cross-modal shared knowledge from task-specific representations. We further design a unified instance encoder, multimodal feature alignment module, and task-aware tracking decoder to enable joint multimodal–multitask optimization. Evaluated on 18 mainstream benchmarks, our method achieves state-of-the-art performance, significantly improving cross-modal transferability and multitask generalization. This work establishes a new paradigm for universal visual modeling.

To address the high cost and low efficiency of manual annotation in video object detection and segmentation, this paper proposes a lightweight, end-to-end automated annotation framework. Methodologically, it introduces the first deep integration of YOLOv8-based video tracking and SAM-based interactive segmentation, augmented by an adaptive frame-sampling strategy and a Gradio-powered visual interface, resulting in a modular, open-source, and extensible prototype system. Key contributions include: (i) a lightweight co-design of tracking and segmentation models; (ii) efficient, interactive annotation support for multi-object, long-duration videos; and (iii) rapid generation of annotated datasets enabling a closed-loop annotation–training pipeline. Experiments on multiple public video benchmarks demonstrate over 10× higher annotation throughput compared to manual labeling, strong inter-annotator consistency, and substantial performance gains for downstream detection and segmentation models trained on the generated data.

This work addresses two key challenges in referring video object segmentation (RVOS): (i) target identification ambiguity—especially in scenes with multiple visually similar objects—and (ii) inconsistent cross-frame mask propagation. To this end, we propose FindTrack, a decoupled framework that uniquely separates target identification from mask propagation. Specifically, it first identifies the target via a vision-language alignment-driven, adaptive keyframe selection mechanism; then, using this keyframe as a reference, it performs independent, reference-guided mask propagation across frames. The method comprises four core components: multimodal feature disentanglement, alignment-aware keyframe selection, reference-guided propagation network, and joint optimization. Extensive experiments demonstrate that FindTrack achieves significant improvements over state-of-the-art methods across multiple benchmarks, delivering enhanced segmentation accuracy and tracking stability—particularly in challenging multi-similar-object scenarios.

Existing video segmentation methods struggle to achieve persistent object discovery and identity consistency in long-duration, ego-motion-rich open-world videos, and conventional 1:1 evaluation protocols fail to fairly assess predictions that are semantically correct but differ in granularity. To address these challenges, this work proposes Savvy—a training-free system that leverages hierarchical mask discovery, delayed admission, and trajectory integration to enable stable object discovery and identity tracking. We introduce the first practical framework for zero-shot long-term open-world video segmentation and propose a granularity-agnostic Open-World Grouping Assessment (OGA) suite, featuring novel metrics such as Identity Persistence (IP) and Identity Concentration (IC). Evaluated on real-world long-duration benchmarks like ScanNet and HM3D, Savvy significantly outperforms strong baselines across STQ, VPQ∞, IP, and IC, demonstrating both methodological efficacy and evaluation robustness.

Existing point tracking models suffer from performance degradation in real-world videos due to domain shift and the scarcity of dense ground-truth annotations. To address this, this work proposes a verifier-guided pseudo-label generation framework that aggregates candidate trajectories from multiple pretrained trackers and employs a learnable meta-verifier to dynamically assess the reliability of per-frame predictions. High-quality pseudo-labels selected by the verifier are then used for self-supervised fine-tuning. This approach substantially improves both pseudo-label quality and data utilization efficiency, achieving state-of-the-art performance on four real-world benchmarks while requiring significantly less fine-tuning data compared to existing self-training methods.

This work addresses the inefficiency and limited scalability of conventional video models that rely on fixed patch-based tokenization, which generates excessive redundant tokens. Unlike existing trajectory-based tokenization methods requiring complex, task-agnostic external segmentation and tracking pipelines, we propose TrajTok—the first end-to-end trainable video tokenizer. TrajTok implicitly clusters spatiotemporal pixels to produce semantically adaptive object trajectory tokens in a single forward pass, decoupling token count from video duration. Token generation is driven by semantic complexity, enabling dynamic granularity control and seamless integration into downstream architectures such as TrajViT2, TrajAdapter, and TrajVLM. Experiments show that TrajViT2, trained from scratch, achieves state-of-the-art accuracy on classification and retrieval tasks while matching the efficiency of advanced compression methods, with particularly pronounced advantages in long-form video understanding.

This work addresses the issue of model drift in multi-frame visual tracking, which commonly arises from naively aggregating noisy historical predictions. To mitigate this, the authors propose DTPTrack, a lightweight and general-purpose module comprising a Temporal Reliability Calibrator (TRC) and a Temporal Guidance Synthesizer (TGS). The TRC dynamically evaluates the reliability of past tracking states, while the TGS leverages these assessments to generate high-quality temporal priors that guide the tracker robustly against drift. Designed for seamless integration, DTPTrack is readily compatible with mainstream architectures such as OSTrack, ODTrack, and LoRAT. Extensive experiments demonstrate its effectiveness, achieving a 77.5% Success rate on LaSOT and an 80.3% AO on GOT-10k, thereby significantly enhancing tracking robustness and setting new state-of-the-art results across multiple benchmarks.

This work proposes a novel end-to-end multi-object tracking approach that circumvents the limitations of traditional methods, which rely on separate detection and association modules and struggle to maintain identity consistency under occlusions due to complex external identity management. For the first time, the authors directly employ a large-scale text-to-video diffusion model (22B-parameter LTX-2.3) as a tracker, applying lightweight in-context LoRA fine-tuning to generate ID-map videos with temporally consistent identity colors. Identities are encoded implicitly at the pixel level, eliminating the need for explicit detection or association components. A windowed chain-generation strategy conditions each subsequent video window on the last frame of the previous one to preserve identity continuity. The method achieves 40.3 HOTA and 44.1 AssA on DanceTrack, substantially outperforming existing approaches, and successfully recovers identities in 42% of 383 occlusion events—dramatically surpassing appearance-embedding baselines, which exhibit near-zero recovery rates.