TimePLE: Rethinking Temporal Representation for Video Temporal Grounding

2026-07-26

AI Summary

This work addresses the limitations of existing video temporal localization methods, which rely on endpoint prediction and struggle to explicitly model the validity, duration, and interval-level semantics of temporal segments. The authors propose TimePLE, a novel approach that reformulates the task as an interval-native joint distribution prediction, directly modeling valid intervals in a position–duration squared space and generating continuous boundaries via a single <|TIMESPAN|> latent state. Key innovations include a geometrically consistent position–duration representation, a duration-aware coordinate correction mechanism, and implicit interval-anchor alignment through a vision-language model. The study also introduces a large-scale, human-verified dataset for precise full-event alignment. Evaluated across four benchmarks, TimePLE achieves an average mIoU of 58.9%, substantially outperforming endpoint-based baselines, with notable gains on short and medium-duration events.

Abstract

Video temporal grounding (VTG) aims to localize the continuous video interval described by a natural-language query. However, current VLM-based methods typically produce this interval indirectly through two endpoint outputs, represented either as discrete timestamp tokens or continuous boundary coordinates. These formulations differ in how endpoints are encoded, but not in what is predicted: the event interval remains a derived object, while interval validity, duration, and interval-level similarity are handled only implicitly. We propose TimePLE, which reformulates VTG from endpoint prediction to interval-native grounding by predicting a single joint distribution over valid temporal intervals. TimePLE maps each interval to a point in a canonical position-duration square, where every support point corresponds to a valid span and neighboring points represent geometrically similar intervals. Given a video and query, the VLM generates a single latent <|TIMESPAN|> token whose hidden state is decoded into a joint interval distribution, refined through duration-aware coordinate correction, and converted into continuous boundaries. The same interval representation is used to encode input temporal anchors, aligning video-side temporal evidence with output-side span prediction. To reliably align the latent span representation with complete event intervals, we curate 90K-scale grounded samples and human-verify 3K-scale benchmark annotations. Experiments across four VTG benchmarks show that TimePLE consistently outperforms endpoint prediction baselines, achieving an average mIoU of 58.9, with clear gains on short-duration and medium-duration events.

Problem

• Video Temporal Grounding
• Temporal Representation
• Interval Prediction
• Vision-Language Models
• Temporal Localization

Innovation

• interval-native grounding
• joint interval distribution
• temporal representation
• video temporal grounding
• duration-aware correction