Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs

2026-10-04

AI Summary

This study addresses the limitation in vision-language models where standard spatial question-answering relies heavily on visual priors, yielding weak gradient signals for geometric features and hindering effective fusion. To overcome this, we propose a novel-view semantic rendering auxiliary task that introduces novel-view prediction—simulating human mental rotation—as a training signal to enforce joint modeling of geometric visibility and visual semantics. By leveraging pretrained 3D models for geometric feature extraction alongside visual encoders and language models, our approach enables end-to-end multimodal auxiliary supervised learning. This work transcends the bottleneck of marginal gains typically observed in conventional geometric feature fusion, substantially enhancing multi-hop spatial reasoning capabilities. Empirical evaluations demonstrate performance improvements of 1.6, 2.2, and 2.9 points on VSI-Bench, ReVSI, and a custom-built dataset, respectively, surpassing existing open-source methods.

Abstract

Recent works augment Vision-Language Models with geometry features from pretrained 3D models, expecting that the geometric signal will boost spatial reasoning. However, we find that simply fusing geometry features and training on standard spatial QA yields only marginal improvements on high-level multi-hop tasks. We attribute this gap to a training-signal problem: standard spatial QA can be largely answered from visual features and language priors, so the geometry pathway receives weak gradients and fails to integrate with the visual features. To provide a training signal that requires geometry, we propose \textbf{novel-view semantic rendering} as an auxiliary training task that requires the model to predict the semantic layout of an unobserved viewpoint, inspired by humans'ability to mentally simulate novel viewpoints during spatial reasoning. This task encourages joint use of both pathways: geometry provides pose-dependent visibility, while vision provides semantic content. Our auxiliary task yields consistent improvements over the geometry-augmented baseline across all three benchmarks (up to +1.6 on VSI-Bench, +2.2 on ReVSI, +2.9 on our 3D-Point-QA dataset) and our full model surpasses prior open-source methods on VSI-Bench and on ReVSI. Project page: https://yuqunw.github.io/Render2Reason/.

Problem

• Vision-Language Models
• Spatial Reasoning
• Geometry Features
• Training Signal
• Novel-View Prediction

Innovation

• Novel-View Semantic Rendering
• Vision-Language Models
• Spatial Reasoning
• Geometry-Augmented VLMs
• Auxiliary Training Task