Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation

2026-06-04

AI Summary

This work addresses the modality gap between perception and reasoning in visual spatial planning, which hinders the recovery of latent state structures from pixels and impedes multi-step planning. The authors propose MGSD, a novel framework that introduces, for the first time, a modality-gap-aware self-distillation mechanism. It first establishes reliable visual state representations through cold-start training and then leverages symbolic states to supervise prefix rollbacks of visual policies, enabling efficient planning from pure visual inputs. During training, symbolic information enhances the visual model’s state reconstruction and path reasoning capabilities, while at inference time, the method operates entirely without symbolic inputs. Experiments demonstrate that MGSD improves the macro-averaged visual planning metric by 19.3% and 18.4% on 4B and 8B models, respectively, substantially narrowing the performance gap with the symbolic-input upper bound.

Abstract

While vision-language models excel at general multimodal understanding, they still struggle with visual spatial planning. We attribute this to a perception-reasoning modality gap: visual planning requires models to infer latent state structures from pixels and then reason over the recovered structure to produce valid actions, whereas symbolic planning directly leverages explicit objects and constraints. This creates dual bottlenecks in visual state recovery and multi-step planning. To address this, we propose MGSD, a two-stage modality-gap-aware self-distillation framework. First, a cold-start grounding stage equips the visual student with reliable state representations, minimizing early perception noise. Second, a privileged teacher transfers planning capabilities via on-policy distillation, using explicit symbolic states to supervise the student's own visual rollout prefixes. Crucially, symbolic data is used strictly during training, leaving inference purely visual. Experiments on visual planning benchmarks show that MGSD consistently improves visual planning across both 4B and 8B backbones, raising the macro average by 19.3% and 18.4%, respectively. The resulting models narrow the gap to symbolic-input upper bounds, while ablations and diagnostics confirm that the improvement comes from both visual state recovery and optimal-path reasoning. These results suggest that modality-gap-aware self-distillation improves not only how models perceive actionable states, but also how they plan over the inferred structure. Code is available at https://github.com/Oranger-l/MGSD.

Problem

• visual spatial planning
• modality gap
• state recovery
• symbolic planning
• perception-reasoning

Innovation

• modality-gap-aware
• self-distillation
• visual spatial planning
• symbolic grounding
• on-policy distillation