Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Tracking

2026-08-01

AI Summary

This work addresses critical limitations in existing visual tracking methods—such as weak semantic discrimination, coarse foreground perception, instance confusion, and lack of error correction—which hinder robust long-term performance and generalization to multimodal scenarios. To overcome these challenges, the authors propose ACTrack, a novel agent-coordinated, tool-augmented framework that integrates heterogeneous foundation models (e.g., SAM3 and vision-language models) as callable tools. Through an event-triggered mechanism, ACTrack dynamically orchestrates complementary modules including instance matching, motion priors, distractor detection, and large language model–based re-prompting, enabling unified multimodal tracking. The framework features a principled tool-triggering and collaboration protocol alongside a parameter-efficient sharing strategy, transcending the constraints of monolithic architectures. Experiments demonstrate that ACTrack significantly outperforms state-of-the-art trackers on eight RGB benchmarks and achieves substantial gains on multimodal datasets—including LasHeR, VisEvent, TNL2K, and DepthTrack—with only 30% trainable parameters.

Abstract

Most current visual trackers adopt a matching-based architecture trained exclusively on tracking datasets, whose performance gains depend heavily on the length of the input context, and have now reached a bottleneck. While high-performance tracking increasingly relies on foundation models, existing methods use them monolithically, adapting a foundation model into a tracker or modify a segmentation foundation model into a tracking pipeline, which fails to exploit complementary strengths. Matching-based trackers excel at instance-level correspondence but lack semantic discrimination and fine-grained foreground perception, whereas segmentation foundation models produce precise masks yet struggle with instance discrimination and multimodal extension. Both paradigms also lack error-correction capabilities for long-term tracking. To address these issues, we propose ACTrack, an agentic coordination framework that treats heterogeneous models as invocable tools under an event-triggered mechanism. ACTrack coordinates a Tracker-based Instance Matching Tool for target discrimination, a SAM3 Motion Tool for mask-derived motion priors, a SAM3 Perception Tool for detecting distractors and instance-conflict cues, and a VLM Reprompt Tool activated only under persistent conflict to mitigate error accumulation. We design a complete tool-invocation trigger mechanism and an inter-tool coordination mechanism, enabling the complementary strengths of different model tools to be fully integrated. Experiments show that ACTrack substantially surpasses the strongest and the largest trackers on eight RGB benchmarks. Furthermore, a parameter-efficient adaptation strategy enables parameter sharing and reuse across tools, achieving unified multimodal tracking with only 30\% trainable parameters while substantially outperforming prior methods on multimodal benchmarks such as LasHeR, VisEvent, TNL2K, and DepthTrack.

Problem

• visual tracking
• foundation models
• multimodal tracking
• error correction
• instance discrimination

Innovation

• agentic coordination
• foundation models
• multimodal visual tracking
• tool invocation
• parameter-efficient adaptation