Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering

2026-08-04

AI Summary

This work addresses the inefficiency of video question-answering models caused by excessive reasoning, noting that many questions can be answered by merely localizing key visual evidence. The authors propose DyLaR (Dynamic Latent Reasoning), a framework that decouples perception and reasoning to enable on-demand inference: it first employs perception latent variables aligned with visual evidence to identify critical frames, then uses an adaptive routing mechanism to determine whether to activate reasoning latent variables grounded in annotated rationales. The approach integrates continuous latent state encoding, rationale distillation, and a reinforcement learning–driven routing policy within a multimodal large language model backbone. Evaluated across nine video benchmarks and four model variants, DyLaR achieves substantial gains in average accuracy while reducing response length to fewer than 20 tokens—for instance, boosting accuracy from 54.0% to 58.2% and shrinking output length from 1220.7 to 18.5 tokens on Qwen3-VL-4B.

Abstract

Video question answering requires models to ground language queries in visual evidence and, when necessary, reason over that evidence across time. Existing methods typically rely on long textual chain-of-thought rationales, even though many questions can be answered as soon as the relevant object, action, or frame is localized. We propose Dynamic Latent Reasoning (DyLaR), which first grounds a question in a short block of perception latents (continuous hidden states that encode query-relevant visual evidence), and then adaptively decides whether to append reasoning latents (continuous thoughts that reason over this evidence in latent space) before answering. DyLaR learns this behavior by grounding perception latents in verified visual evidence and distilling verified rationales into reasoning latents, followed by reinforcement learning that further refines when to reason. Across nine video benchmarks and four multimodal language model backbones, DyLaR improves average accuracy over same-backbone baselines while generating fewer than 20 tokens per query. On Qwen3-VL-4B, for example, DyLaR improves average accuracy over Qwen3-VL-4B-Thinking from 54.0 to 58.2 while reducing response length from 1,220.7 to 18.5 tokens per query. Ablations further show that grounded perception latents, rationale-supervised reasoning latents, and adaptive routing each improve accuracy.

Problem

• video question answering
• perception
• reasoning
• multimodal language models
• efficiency

Innovation

• Dynamic Latent Reasoning
• Perception Latents
• Reasoning Latents
• Video Question Answering
• Adaptive Routing