ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA

2026-07-30

AI Summary

Existing 3D question answering (3D-QA) methods rely heavily on costly 3D annotations, limiting their scalability. This work proposes the first training-free, modular 3D-QA framework that integrates a general-purpose large language model (LLM) and a vision-language model (VLM) through four synergistic components: multi-view selection, language-guided visual grounding, bird’s-eye-view (BEV) spatial encoding, and role-based reasoning. This design enables interpretable, robust performance with a strong capability for spatial reasoning. Evaluated on ScanQA and SQA3D, the method achieves an overall accuracy of 50.8% and a CIDEr score of 73.4, substantially outperforming existing training-free approaches—particularly excelling on spatial-relation questions.

Abstract

Recent advances in large language models (LLMs) and vision-language models (VLMs) have enabled new possibilities for 3D question answering (3D-QA), a key capability for embodied AI and robotic perception. However, most existing methods rely on 3D-specific training or fine-tuning with costly annotations, limiting their scalability and real-world applicability. We present \textbf{ViewMind3D}, a fully training-free and modular framework for 3D spatial reasoning over multi-view observations of a scene without requiring complete 3D reconstruction. The framework decomposes the 3D-QA task into four interpretable components: (1) question-driven multi-view selection, (2) guided visual grounding with language-conditioned object cues, (3) spatial context encoding via a bird's-eye-view (BEV) viewpoint indicator, and (4) structured answer generation through role-based reasoning. This design enables structured, robust, and interpretable reasoning without requiring model tuning. Experimental results on ScanQA and SQA3D show that ViewMind3D achieves competitive performance compared to prior training-free and fine-tuned 3D-LLMs. In particular, our method improves performance on spatially grounded question types, such as ``What'' questions in SQA3D, while maintaining strong overall accuracy (50.8\%) and achieving 73.4 CIDEr on ScanQA. These results demonstrate that effective 3D reasoning can be achieved through modular orchestration of general-purpose LLMs and VLMs for robotic perception in real-world environments.

Problem

• 3D question answering
• training-free
• embodied AI
• robotic perception
• scalability

Innovation

• training-free
• modular reasoning
• 3D question answering
• view-aware inference
• spatial grounding