Track4Action: Distilling World-Centric 3D Tracker into Vision-Language-Action Policies

2026-08-04

AI Summary

This work addresses the limitation of existing vision-language-action (VLA) policies, which lack explicit supervision over the 3D world dynamics induced by actions and struggle to model geometric, motion, and viewpoint changes using only action labels. To overcome this, the authors introduce a frozen world-centric 3D tracker that distills multi-frame transition features from demonstration videos as privileged supervisory signals. These features are matched via learnable trajectory queries and integrated into the VLA policy through a feature-gated flow-matching action head and a shared spatial alignment mechanism. Notably, the tracker and historical video segments are discarded at deployment. The method achieves state-of-the-art performance, attaining 82.3% success on LIBERO-Plus zero-shot tasks (+7.6%), 80.44% and 81.48% on RoboTwin 2.0 cleaning and random splits, respectively, and an average real-world success rate of 67.5% across four bimanual tasks—surpassing baselines by 25.0 percentage points.

Abstract

Action labels tell a vision-language-action (VLA) policy which robot commands to imitate, but not how those commands change the 3D world. The aligned demonstration clip contains this missing supervision because its $K$ frame transitions record the geometry, motion, visibility, and camera change produced during the corresponding $K$ actions. We introduce Track4Action, a framework that distills this realized transition from a frozen world-centric 3D tracker into a current-observation VLA policy. During training, Track4World encodes the clip $V_{t:t+K}$ into a pooled tracker feature. Learnable track queries infer this feature from current VLA hidden states, match it in a shared space, and condition a flow-matching action head through a feature-wise gate. The tracker feature only defines the alignment target, so neither the clip nor the tracker is used at deployment. Track4Action reaches 82.3% on zero-shot LIBERO-Plus, improving the alignment-free variant by 7.6 points and LaMP by 3.0 points. It obtains 80.44% and 81.48% on the clean and randomized RoboTwin 2.0 splits, and 67.5% average success across four physical bimanual tasks, 25.0 points above the alignment-free variant. The gains across simulation and physical tasks support action-aligned 3D tracker features as privileged supervision for tracker-free VLA deployment. Our project page is available at https://wing0night.github.io/track4action-project-page.

Problem

• vision-language-action
• 3D tracking
• action alignment
• robotic policy
• world-centric supervision

Innovation

• vision-language-action
• 3D tracking distillation
• action-aligned supervision
• tracker-free deployment
• flow-matching policy