Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding

2026-07-17

AI Summary

This work addresses the tension between limited visual token budgets and the need for precise capture of critical events in long-form video multi-event understanding, as well as the lack of adaptive allocation and self-correction capabilities in existing two-stage approaches. To this end, we propose MoD-VLLM, a novel framework featuring a modular dynamic granularity mechanism and a reflexive closed-loop architecture. By iteratively co-optimizing temporal localization and semantic understanding through positive-negative segment localization and a dynamic granularity reflection module, MoD-VLLM enables fine-grained encoding of relevant segments and coarse-grained compression of irrelevant ones. A reinforcement learning strategy is further introduced to jointly optimize localization and representation. Extensive experiments on multiple long-video benchmarks and the newly constructed MEventBench demonstrate that MoD-VLLM significantly outperforms state-of-the-art methods, validating its effectiveness in complex multi-event reasoning.

Abstract

Video Large Language Models (Video LLMs) have made significant advancements in various video understanding tasks. However, long-video scenarios remain challenging due to the tension between limited visual token budgets and the need to capture multiple key events. Existing approaches typically process long videos in two stages, i.e., i) select keyframes and ii) perform detailed perception, which exhibit limitations: they lack a modular mechanism for adaptive capacity allocation and self-correction, resulting in unreliable modeling. To tackle these challenges, we propose MoD-VLLM, a novel Modularized Dynamic-Granularity Video LLM framework for multi-event long video understanding, which unifies temporal grounding and semantic understanding iteratively and self-reflectively. Specifically, we propose a Positive-Negative Video Segments Grounding module and a Modularized Dynamic-Granularity Reflection module, which form a closed loop to progressively localize the question-related video segments. The grounding module instructs a Video LLM to distinguish relevant from irrelevant video segments based on the video question. The reflection module employs a modularized scheduler that dynamically selects fine-grained encoding for relevant positive segments to capture detailed perception and coarse-grained encoding for negative segments to maintain global context. We further propose a dynamic-granularity reinforcement learning strategy, allowing MoD-VLLM to learn optimal grounding policies and dynamic granularity visual representation jointly. Moreover, we propose MEventBench, a challenging Multi-Event Long Video Benchmark for complex long video reasoning. Extensive experiments on several long video understanding benchmarks and our MEventBench demonstrate that MoD-VLLM significantly outperforms state-of-the-art baselines.

Problem

• long video understanding
• multi-event reasoning
• visual token budget
• modular mechanism
• adaptive capacity allocation

Innovation

• Modularized Dynamic-Granularity
• Video LLM
• Temporal Grounding
• Reinforcement Learning
• Multi-Event Long Video Understanding