T-STAR: A Large-Scale Benchmark for Spatio-Temporal Panoptic Scene Graph Generation in Satellite Video

2026-07-23

AI Summary

This work addresses the absence of dedicated datasets and effective methods for spatio-temporal panoptic scene graph generation (TPSG) in satellite video, where challenges such as small objects, occlusions, and complex spatio-temporal relationships remain unresolved. We introduce the TPSG task, which jointly models identity-consistent instance masks and spatio-temporal relations among panoptic elements through time-aware <subject, relation, object> triplets with explicit temporal spans. To support this task, we present T-STAR, the first large-scale satellite video benchmark for TPSG, comprising 1.1 million instance masks and 3.8 million spatio-temporal triplets across 39 object and 70 relation categories. Furthermore, we propose a unified end-to-end framework that integrates instance segmentation, cross-frame association, and spatio-temporal relation reasoning, substantially advancing structured scene understanding and establishing foundational data and methodological resources for the field.

Abstract

Structured understanding of satellite video is essential for advancing dynamic geospatial scene analysis from low-level perception to high-level cognition. To move beyond object-centric perception, this paper introduces spatio-temporal panoptic scene graph generation (TPSG) in satellite video as a new benchmark task. TPSG aims to generate a structured graph composed of a set of triplets <subject, relationship, object> with explicit temporal spans, thereby describing dynamic geospatial scenes by jointly modeling identity-consistent instance masks and spatio-temporal relationships among panoptic scene elements. However, there is still no dedicated dataset for TPSG in satellite video. Moreover, TPSG in satellite video is intrinsically challenging, as objects are often small and weakly textured, cross-frame association is easily disrupted by occlusion and background clutter, and relationship semantics are highly coupled with spatial structure and temporal evolution. Consequently, TPSG models developed for natural videos are not directly applicable to satellite video. This paper presents T-STAR, a large-scale benchmark dataset for TPSG in satellite video, comprising over 1.1 million instance masks and over 3.8 million spatio-temporal triplets across 39 fine-grained object categories and 70 fine-grained relationship categories. To enable TPSG in satellite video, we propose a unified framework to enhance cross-frame instance consistency and spatio-temporal relationship prediction. Extensive experiments demonstrate the significance of T-STAR and the effectiveness of the proposed framework, establishing a strong benchmark for future research on structured satellite video understanding. The dataset and code are available at https://github.com/linlin-dev/T-STAR.

Problem

• spatio-temporal panoptic scene graph
• satellite video
• structured scene understanding
• benchmark dataset
• dynamic geospatial scenes

Innovation

• spatio-temporal panoptic scene graph
• satellite video understanding
• T-STAR benchmark
• cross-frame instance consistency
• structured scene representation