Visual Saliency Steering Distillation for Multimodal Chain-of-Thought Reasoning

2026-07-24

AI Summary

This work addresses the challenge that small models struggle to discern subtle semantic variations in text–image pairs during multimodal chain-of-thought reasoning, primarily due to modality fusion suppressing fine-grained cross-modal differences. To mitigate this, the authors propose a visual saliency–guided distillation mechanism: task-sensitive perturbed images are generated using attention maps from a large multimodal language model, and dominant guidance vectors are extracted via singular value decomposition to enable layer-wise cross-modal knowledge distillation. This approach effectively preserves and amplifies critical cross-modal discrepancy signals, substantially enhancing the reasoning capabilities of compact models under limited token budgets. Empirical results on ScienceQA and M³CoT benchmarks demonstrate consistent improvements in both answer accuracy and the quality of generated reasoning rationales.

Abstract

Multimodal chain-of-thought (CoT) reasoning integrates visual and textual cues through step-by-step inference. In small models with limited token budgets, modality-interaction fusion often suppresses tiny cross-modal differences. In particular, multimodal CoT often struggles when different images pair with identical text or different texts pair with an identical image, making such inputs nearly indistinguishable after fusion. This study proposes Visual Saliency Steering Distillation (VSSD). VSSD leverages the attention maps of multimodal large language models to generate perturbed images that capture task-sensitive feature directions, and then applies singular value decomposition to extract dominant steering vectors to guide inter-layer distillation. Experiments on ScienceQA and M$^3$CoT demonstrate that VSSD improves rationale generation and answer inference. The code is available at https://github.com/BGWH123/VSSD.

Problem

• multimodal chain-of-thought
• visual saliency
• cross-modal differences
• modality-interaction fusion
• token budget

Innovation

• Visual Saliency Steering Distillation
• Multimodal Chain-of-Thought
• Attention-guided Perturbation
• Singular Value Decomposition
• Inter-layer Distillation