ReferTrack: Referring Then Tracking for Embodied Visual Tracking

2026-07-22

AI Summary

This work addresses the challenge of misalignment between target reference and trajectory planning in embodied visual tracking under monocular vision and natural language instructions. The authors propose a “reference-then-track” paradigm: first explicitly selecting the linguistically described target in image space, then decoding a tracking trajectory based on this decision. Their method introduces Temporal View Bounding Indicator (TVBI) tokens to fuse geometric features, incorporates a sliding-window memory mechanism to retain historical motion cues, and leverages a newly constructed Refer-QA dataset for joint training of vision–language–action policies. Evaluated on the EVT-Bench single-view setting, the approach achieves state-of-the-art success rates of 89.4%, 73.3%, and 74.1% across three task categories—matching or surpassing multi-camera baselines—and demonstrates strong sim-to-real transferability on both quadrupedal and humanoid robots.

Abstract

Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.

Problem

• embodied visual tracking
• referring expression
• target identification
• visual grounding
• single-view tracking

Innovation

• referring-then-tracking
• temporal-viewpoint-bbox indicator (TVBI)
• embodied visual tracking
• vision-language-action (VLA)
• image-grounded decision