TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

2026-07-19

AI Summary

This work addresses the limited generalization of existing video multimodal large models in temporally localizing events across varying video lengths, domains, query formulations, and viewpoints. To overcome this, we propose TimeLens2, a framework that models temporal evidence as sets of intervals and preserves set structure throughout the pipeline. It employs a multi-stage process to generate reliable multi-span supervision signals and introduces a matching-agnostic dense reward mechanism. Innovatively, TimeLens2 incorporates a Wasserstein distance–based temporal reward function and a cross-agent consensus mechanism, enabling set-level supervision without explicit segment matching. Integrated into the Qwen3-VL architecture, the full pipeline combines caption-guided candidate generation, independent localization, semantic verification, and boundary refinement. TimeLens2-2B outperforms same-scale baselines on all seven benchmarks, while its 4B and 8B variants achieve state-of-the-art results, improving mIoU by 13.0 and 18.1 points over the original Qwen3-VL—surpassing even a 397B-parameter open-source model.

Abstract

Video multimodal large language models (MLLMs) can describe what happens in a video, but rarely identify when the supporting evidence occurs. We study generalist video temporal grounding, in which one model predicts a variable-cardinality set of evidence intervals across video lengths, domains, query forms, and viewpoints. Existing training strategies are misaligned with this set-valued task: long-video labels often rely on brittle one-pass annotation, while reinforcement-learning rewards either fail to distinguish non-overlapping predictions or require fragile segment matching. TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization. TimeLens2-93K constructs reliable multi-span supervision through caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. Our temporal Wasserstein reward computes exact one-dimensional \(W_1\) between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities and equivalent fragmentation; temporal IoU complements it with precise-overlap feedback. Across seven benchmarks, TimeLens2-2B outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively.

Problem

• video temporal grounding
• multimodal LLMs
• temporal evidence localization
• interval prediction
• generalist video understanding

Innovation

• temporal grounding
• multimodal LLMs
• Wasserstein reward
• interval set supervision
• video understanding