TBSG-Net: Temporal Bipartite Scene Graph Network for Fine-Grained Video Moment Retrieval

2026-08-03

AI Summary

This work addresses the limitations of existing video moment retrieval methods that rely on static scene graphs, which fail to capture the temporal dynamics and durations of object relationships, leading to inadequate representation of critical temporal dependencies and suboptimal localization accuracy. To overcome this, the paper proposes TBSG-Net—the first candidate-free retrieval model based on dynamic scene graphs—introducing a novel Temporal Bipartite Scene Graph (TBSG) to explicitly model objects, their relationships, and associated time spans. The framework incorporates a TBSG constructor and a hybrid encoder that synergistically combines Transformers with Graph Convolutional Networks to enable multi-level spatiotemporal modeling and fine-grained relational reasoning. Extensive experiments demonstrate that the proposed method significantly outperforms state-of-the-art approaches across multiple benchmarks, confirming the effectiveness and superiority of dynamic scene graphs for video moment retrieval.

Abstract

Recent advances in proposal-free Video Moment Retrieval (VMR) have highlighted the effectiveness of Static Scene Graphs (SSGs). By modeling objects and their relations at the frame level, SSGs enrich retrieval-oriented video representations. However, integrating SSGs into VMR remains constrained by two inherent limitations: (1) Lack of Temporal Dynamics. SSGs fail to model how objects and their relationships evolve over time, leading to the loss of essential temporal dependencies in video representation; and (2) Lack of Explicit Temporal Span Encoding. SSGs do not explicitly encode the duration of relationships, making precise localization challenging. To address these limitations, we propose Temporal Bipartite Scene Graph Network (TBSG-Net)---to the best of our knowledge, the first Dynamic Scene Graph (DSG) based proposal-free VMR model. Specifically, TBSG-Net leverages DSGs to extract event-centric graph representations of the input video, enabling the modeling of object interactions over time and thus addressing limitation (1). These DSGs are then processed by a novel Dynamic Scene Graph Embedding (DSG-E) module to capture both Temporal Span and spatio-temporal information. First, DSG-E utilizes a TBSG Constructor to transform DSGs into TBSGs, explicitly encoding objects, relationships, and time spans to tackle limitation (2). Second, the resultant TBSGs are passed into a hybrid TBSG Encoder that integrates a Transformer variant for global event modeling and a Graph Convolutional Network for detailed relational reasoning, ultimately producing a more comprehensive spatio-temporal representation. Our experiments demonstrate substantial improvements of TBSG-Net over all baselines.

Problem

• Video Moment Retrieval
• Temporal Dynamics
• Scene Graph
• Temporal Span Encoding
• Proposal-free

Innovation

• Temporal Bipartite Scene Graph
• Dynamic Scene Graph
• Video Moment Retrieval
• Proposal-free
• Spatio-temporal Representation