Distill What RGB Can Recover: Privileged 3D Evidence for RGB-Only Vision-Language Models

2026-07-31

AI Summary

This work addresses the limitations of existing RGB-only methods, which struggle with complex scene understanding due to the absence of 3D cues and often rely on depth or 3D information during inference, hindering practical deployment. To overcome this, the authors propose an evidence-aware knowledge distillation framework that leverages privileged 3D evidence during training to transfer spatial reasoning capabilities from a teacher model to an RGB-only student model. The approach employs a unified evidence interface, controlled residual injection, and joint distillation of logits and structured representations. Innovatively integrating recoverability decomposition analysis with evidence perturbation evaluation, the method effectively distinguishes between RGB-recoverable information and the teacher’s irreducible advantage, thereby avoiding the imitation of unreliable signals. Experiments show the student model surpasses baselines across all 11 metrics, achieving CIDEr@0.5 gains of 10.4 on ScanQA and 19.1 on Scan2Cap, while the teacher attains state-of-the-art performance on 7 metrics.

Abstract

3D scene understanding requires reasoning about entity existence, spatial layout, and object relations, yet RGB images alone often provide insufficient 3D cues. Existing 3D-VLMs commonly rely on depth or 3D-position-aware inputs at inference time, introducing additional acquisition, reconstruction, or annotation costs that limit RGB-only deployment. We therefore study how training-time 3D evidence can be converted into spatial reasoning capabilities retained under RGB-only inference. We propose a privileged-evidence distillation framework that constructs a distillable teacher through a unified evidence interface and controlled residual injection, and transfers its knowledge to a deployable student receiving only RGB images and questions through logit and structured representation distillation. To avoid imitating teacher signals unsupported by RGB, we further introduce evidence-sensitivity-guided distillation, which uses corrupted evidence to identify highly evidence-dependent targets and down-weight their supervision. We also define a recoverability decomposition based on the matched baseline, teacher, and student, separating privileged gains into RGB-recoverable improvements and residual teacher advantages. Across four benchmarks, the teacher achieves the best result on 7 of 11 reported metrics among the compared methods. The RGB-only student outperforms its matched baseline on all 11 metrics, including gains of 10.4 ScanQA CIDEr and 19.1 Scan2Cap CIDEr@0.5, without additional inference-time inputs. These results validate the effectiveness of training-time privileged 3D evidence distillation for both teacher performance and deployable RGB-only spatial reasoning. Separately, our matched baseline-teacher-student analysis characterizes privileged-gain transfer across evidence types and spatial skills.

Problem

• 3D scene understanding
• RGB-only inference
• vision-language models
• privileged distillation
• spatial reasoning

Innovation

• privileged distillation
• RGB-only vision-language models
• 3D scene understanding
• evidence-sensitivity-guided distillation
• recoverability decomposition