AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos

2026-09-21

AI Summary

本文提出了一种通过分析-合成方法从单目视频中进行形状跟踪和重建的方法，使用视觉-语言模型代理迭代优化物体的3D模型和姿态估计。

Abstract

In this work, we present a method for shape reconstruction and tracking from video via agentic analysis-by-synthesis. Unlike prior methods which first estimate dense pixel correspondences and then recover object motion from them, our method infers a structured 3D object model, including its geometry and kinematic structure, and uses this model to optimise object track estimates over time. In our optimisation loop, a Vision-Language Model (VLM) agent iteratively refines shape or generalised pose through a render-and-compare loop, combining coarse visual reasoning with numerical pose optimisation for precise state estimation. This structured formulation enables our method to track through large motion, articulation, and severe occlusion without relying on pixel-matching objectives. Quantitatively, on ARCTIC, our method substantially outperforms state-of-the-art 3D point-tracking baselines for articulated objects, and on HOT3D it outperforms all evaluated rigid-object tracking baselines.

Problem

• shape reconstruction
• tracking
• monocular video
• articulation
• occlusion

Innovation

• agentic analysis-by-synthesis
• structured 3D object model
• render-and-compare loop
• vision-language model