Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

2026-05-31

AI Summary

Existing foundation models struggle to reproduce target image viewpoints through active exploration in 3D environments, primarily due to their reliance on passive perception and static scene understanding. This work introduces the Target Viewpoint Reproduction (TVR) task and the TVRBench benchmark, formally framing viewpoint reproduction as an embodied active perception problem and revealing critical deficiencies in models’ multi-turn visual memory and spatial displacement reasoning. To address these limitations, we develop a unified post-training framework that integrates expert trajectory supervision, chain-of-thought supervision, and multi-turn GRPO optimization. Experiments demonstrate that our approach substantially improves the success rate of a 9B open-source model on TVRBench from 7.8% to 51.4%, significantly enhancing its spatial intelligence and embodied action capabilities, thereby establishing a new evaluation platform and training paradigm for embodied foundation models.

Abstract

Humans can reproduce the viewpoint specified by a target image through active head and body motion, yet spatial intelligence in foundation models has largely been studied as passive understanding of pre-collected observations. We introduce Target Viewpoint Reproduction (TVR) -- an active task where an agent adjusts its viewpoint in a 3D environment until its observation matches a given target image -- and TVRBench, an indoor-simulation benchmark spanning scene scale and target-view visual richness. TVR is far from solved: on the evaluation split, the strongest open-source and closed-source models reach only 7.8% and 12.0% success. Fine-grained analysis identifies two consistent bottlenecks: off-the-shelf models struggle with multi-turn visual history, and performance drops sharply when viewpoint reproduction requires body translation rather than in-place rotation, exposing a gap in mapping spatial discrepancies to embodied movement. To study reducing this gap, we build a unified TVR post-training framework covering expert-trajectory SFT, rationale-supervised CoT-SFT, offline Single-turn GRPO, and on-policy Multi-turn GRPO from live simulator rollouts. Visual-action SFT supplies the main gain, raising a 9B open-source model to 50.8% success; Multi-turn GRPO provides targeted multi-room refinement and reaches 51.4% overall, while CoT supervision and Single-turn GRPO degrade closed-loop performance. These results establish TVRBench as a testbed for measuring and training foundation models that actively perceive and act in 3D environments. Our code, data, and models are available at https://github.com/aim-uofa/TVRBench.

Problem

• Target Viewpoint Reproduction
• active exploration
• foundation models
• 3D environment
• spatial intelligence

Innovation

• Target Viewpoint Reproduction
• Active Exploration
• Embodied AI
• Foundation Models
• 3D Navigation