Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding

2026-07-30

AI Summary

This work addresses the limitation of existing long video understanding methods, which rely solely on keyframes as explicit evidence and struggle to effectively integrate complementary cross-frame information for complex question answering. To overcome this, we propose GenEvA, a novel framework that formulates evidence aggregation as a query-guided implicit interface. GenEvA dynamically models the evidence distribution conditioned on the input query, generating compact cross-frame implicit evidence and adaptively determining whether to inject it into the answer generation process. The framework is end-to-end trainable with Video-MLLM backbones and achieves significant performance gains across four benchmarks: under an 8-frame setting, it improves LLaVA-Video by 5.2 points on average and boosts Qwen2.5-VL by 10.1 points on LVBench, with only a marginal increase of 0.11%–0.40% in video token overhead.

Abstract

Long-video understanding commonly compresses videos into a small set of frames or visual tokens for answer generation. Existing compact pipelines focus on retaining relevant visual content as explicit evidence. Yet making evidence available does not ensure that complementary cues across moments are integrated for answering. Our key idea is to organize selected frames into query-relevant cross-frame evidence before generation. We formulate this post-selection stage as a latent evidence interface and instantiate it with GenEvA ($\textbf{Gen}erative$ $Latent$ $\textbf{Ev}idence$ $\textbf{A}ggregation$), a distribution-guided latent evidence aggregation framework. Specifically, GenEvA uses a query-conditioned evidence distribution to focus aggregation on relevant frames, forming compact cross-frame latent evidence from their frame-specific information. Since cross-frame integration is not always needed, the same distribution determines whether to insert this latent complement. Across four benchmarks and two Video-MLLM backbones, GenEvA consistently improves matched-frame baselines. At 8 frames, it raises the four-benchmark LLaVA-Video average by $+5.2$ points and Qwen2.5-VL accuracy on LVBench by $+10.1$ points. These gains require only $0.11\%$--$0.40\%$ average video-token overhead; analyses further show task-aware allocation and benefits from Adaptive Evidence Invocation.

Problem

• long-video understanding
• evidence aggregation
• cross-frame integration
• video question answering
• latent evidence

Innovation

• Latent Evidence Aggregation
• Query-Conditioned Distribution
• Cross-Frame Integration
• Adaptive Evidence Invocation
• Long-Video Understanding