Shallow Features Matter: Hierarchical Memory with Heterogeneous Interaction for Unsupervised Video Object Segmentation

2025-07-30

AI Summary

In unsupervised video object segmentation (UVOS), the absence of pixel-level annotations leads to fine-grained information loss, while existing methods over-rely on high-level semantic features and neglect low-level details. To address this, we propose a hierarchical memory architecture that explicitly fuses shallow-layer texture/edge cues with deep semantic representations. We design a pixel–semantic heterogeneous interaction mechanism: a Pixel-guided Local Alignment Module (PLAM) enables fine-grained spatial alignment, while a Semantic-guided Global Integration Module (SGIM) performs contextual aggregation. Our approach further incorporates multi-scale feature fusion and adaptive interaction learning. Evaluated on DAVIS, YouTube-VOS, and video saliency detection benchmarks, it achieves state-of-the-art performance. Moreover, it demonstrates strong robustness and consistency across diverse backbones—including ResNet and ViT—validating its architectural generality and effectiveness.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Unsupervised Video Object Segmentation (UVOS) aims to predict pixel-level masks for the most salient objects in videos without any prior annotations. While memory mechanisms have been proven critical in various video segmentation paradigms, their application in UVOS yield only marginal performance gains despite sophisticated design. Our analysis reveals a simple but fundamental flaw in existing methods: over-reliance on memorizing high-level semantic features. UVOS inherently suffers from the deficiency of lacking fine-grained information due to the absence of pixel-level prior knowledge. Consequently, memory design relying solely on high-level features, which predominantly capture abstract semantic cues, is insufficient to generate precise predictions. To resolve this fundamental issue, we propose a novel hierarchical memory architecture to incorporate both shallow- and high-level features for memory, which leverages the complementary benefits of pixel and semantic information. Furthermore, to balance the simultaneous utilization of the pixel and semantic memory features, we propose a heterogeneous interaction mechanism to perform pixel-semantic mutual interactions, which explicitly considers their inherent feature discrepancies. Through the design of Pixel-guided Local Alignment Module (PLAM) and Semantic-guided Global Integration Module (SGIM), we achieve delicate integration of the fine-grained details in shallow-level memory and the semantic representations in high-level memory. Our Hierarchical Memory with Heterogeneous Interaction Network (HMHI-Net) consistently achieves state-of-the-art performance across all UVOS and video saliency detection benchmarks. Moreover, HMHI-Net consistently exhibits high performance across different backbones, further demonstrating its superiority and robustness. Project page: https://github.com/ZhengxyFlow/HMHI-Net .

Problem

• UVOS lacks fine-grained information without annotations
• Existing methods over-rely on high-level semantic features
• Propose hierarchical memory for pixel-semantic feature integration

Innovation

• Hierarchical memory with shallow and high-level features
• Heterogeneous interaction for pixel-semantic mutual alignment
• Pixel-guided and semantic-guided modules for feature integration