Time2General: Learning Spatiotemporal Invariant Representations for Domain-Generalization Video Semantic Segmentation

2026-02-10

AI Summary

This work addresses the challenges of inter-frame flickering and prediction inconsistency in domain-generalized video semantic segmentation, which arise from domain shifts and variations in temporal sampling. To tackle these issues, the authors propose Time2General, a novel framework that leverages a Spatio-Temporal Memory Decoder to aggregate multi-frame contextual information into a coherent spatio-temporal memory, enabling the decoding of temporally consistent per-frame masks without explicit correspondence propagation. The approach introduces three key innovations: a stability query mechanism, a mask temporal consistency loss, and a random training step-length strategy, collectively enhancing robustness to unseen domains and diverse frame rates while improving temporal stability. Extensive experiments on multiple driving-scene benchmarks demonstrate that Time2General outperforms existing methods in both cross-domain accuracy and temporal consistency, achieving a real-time inference speed of 18 FPS.

Technology Category

• Computer Vision: Segmentation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Domain Generalized Video Semantic Segmentation (DGVSS) is trained on a single labeled driving domain and is directly deployed on unseen domains without target labels and test-time adaptation while maintaining temporally consistent predictions over video streams. In practice, both domain shift and temporal-sampling shift break correspondence-based propagation and fixed-stride temporal aggregation, causing severe frame-to-frame flicker even in label-stable regions. We propose Time2General, a DGVSS framework built on Stability Queries. Time2General introduces a Spatio-Temporal Memory Decoder that aggregates multi-frame context into a clip-level spatio-temporal memory and decodes temporally consistent per-frame masks without explicit correspondence propagation. To further suppress flicker and improve robustness to varying sampling rates, the Masked Temporal Consistency Loss is proposed to regularize temporal prediction discrepancies across different strides, and randomize training strides to expose the model to diverse temporal gaps. Extensive experiments on multiple driving benchmarks show that Time2General achieves a substantial improvement in cross-domain accuracy and temporal stability over prior DGSS and VSS baselines while running at up to 18 FPS. Code will be released after the review process.

Problem

• Domain Generalization
• Video Semantic Segmentation
• Temporal Consistency
• Domain Shift
• Temporal Sampling Shift

Innovation

• Spatio-Temporal Memory Decoder
• Stability Queries
• Masked Temporal Consistency Loss
• Domain Generalized Video Semantic Segmentation
• Temporal Flicker Suppression