MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models

2026-06-10

AI Summary

This work addresses the hallucination problem in video large language models, where generated responses often deviate from the input video content. To enhance response fidelity, the authors propose a multimodal context-aware visual token rectification framework that identifies and replaces unreliable visual tokens prior to language generation. The approach employs a lightweight rectifier coupled with an information-guided ranking calibration mechanism, integrating dynamic global patch replacement, answer-conditioned frame-level guidance, ground-truth answer supervision, and sparse regularization. Notably, this method enables fine-grained correction of local visual evidence without modifying the original model architecture. Experiments demonstrate significant performance gains: it improves the F1 score of Qwen3-VL-4B-Instruct by 50.60% on Vript-HAL and yields an 18.58% relative accuracy improvement for Video-LLaVA-7B on ActivityNet-QA, with negligible inference overhead.

Abstract

Video Large Multimodal Models have achieved remarkable progress in video understanding, yet they remain prone to hallucinations, where generated responses are not faithfully supported by the input video. In this paper, we propose MultiToP, a multimodal-context-aware visual token patching framework that mitigates hallucinations by refining unreliable visual tokens before language generation. MultiToP introduces a lightweight Visual Token Patcher to predict token-level replacement distributions and selectively substitute unreliable visual tokens with a dynamic global patch token. To train the patcher effectively, we further propose information-guided rank calibration, which uses answer-conditioned frame-level information cues derived from the backbone to guide token replacement. Combined with ground-truth answer supervision and sparsity regularization, MultiToP enables localized visual evidence refinement without modifying the original model. Extensive experiments demonstrate that MultiToP effectively reduces hallucinations on Vript-HAL with negligible inference overhead, improving the F1 scores of Qwen3-VL-4B-Instruct by 50.60% over the vanilla model. Meanwhile, MultiToP preserves general video understanding ability, yielding an 18.58% relative accuracy gain on ActivityNet-QA for Video-LLaVA-7B.

Problem

• hallucinations
• video large multimodal models
• visual tokens
• video understanding
• multimodal

Innovation

• visual token patching
• hallucination mitigation
• multimodal-context-aware
• information-guided rank calibration
• video large multimodal models