MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models

2026-09-30

AI Summary

This study addresses the limited multimodal in-context learning capabilities of small models and the reliance on spurious cues inherent in conventional knowledge distillation. To this end, we propose the MCD framework, which pioneers the integration of causal inference into multimodal knowledge distillation. By employing structure-preserving token interventions to identify causal evidence, MCD shifts the student model’s objective from merely imitating output predictions to internalizing the teacher’s multimodal evidence utilization mechanisms. Extensive experiments demonstrate that MCD achieves an average improvement of 7.23 points across seven benchmarks, outperforming standard distillation approaches by 4.68 points. These results effectively validate both the generalizability and superiority of the proposed method.

Abstract

Large vision-language models (LVLMs) exhibit strong multimodal in-context learning (ICL) capabilities, yet this ability degrades substantially as model size decreases. Knowledge distillation offers a natural way to bridge this gap, but existing methods primarily align output distributions or hidden representations directly. Such alignment teaches the student what the teacher predicts without revealing which evidence in the complex context causally supports that prediction. Consequently, a student can imitate the teacher's answer while continuing to rely on language priors, prompt structure, or other spurious cues. To address this limitation, we introduce Multimodal Causal Distillation (MCD), a distillation framework that transfers how a strong teacher uses multimodal evidence during ICL. MCD uses structure-preserving token interventions to identify and verify causal evidence, then transfers how the teacher responds when that evidence is retained or removed. This design connects distillation to the causal patterns by which the model uses contextual evidence during multimodal ICL. Experiments across three LVLM families and seven benchmarks show that MCD improves student performance by 7.23 points on average and outperforms vanilla distillation by 4.68 points, while further analyses confirm the generalizability of these gains.

Problem

• Multimodal In-Context Learning
• Knowledge Distillation
• Large Vision-Language Models
• Causal Evidence
• Spurious Cues

Innovation

• Causal Distillation
• Multimodal In-Context Learning
• Token Interventions
• Knowledge Distillation
• Large Vision-Language Models