Aligning Instance-Semantic Sparse Representation towards Unsupervised Object Segmentation and Shape Abstraction with Repeatable Primitives.

2025-01-21

IEEE Transactions on Visualization and Computer Graphics

AI Summary

This paper addresses the challenge of jointly modeling instance segmentation, semantic segmentation, and part-level shape abstraction in unsupervised 3D shape understanding. To this end, we propose the first single-stage, end-to-end framework. Methodologically, we introduce an instance-semantic dual-level sparse representation alignment mechanism, integrating sparse latent membership tracking, attention-driven feature alignment, and cascaded unfreezing of geometric parameters—thereby unifying semantic disentanglement with geometric reusability. Unlike prior approaches, our method requires no annotations or hand-crafted semantic priors and avoids multi-stage training. On standard benchmarks, it achieves state-of-the-art performance in unsupervised joint segmentation. Moreover, it generates part-level shape abstractions that are cross-category consistent, semantically interpretable, and geometrically reusable.

Technology Category

• Computer Vision: Segmentation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Unsupervised & Self-Supervised Learning

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Understanding 3D object shapes necessitates shape representation by object parts abstracted from results of instance and semantic segmentation. Promising shape representations enable computers to interpret a shape with meaningful parts and identify their repeatability. However, supervised shape representations depend on costly annotation efforts, while current unsupervised methods work under strong semantic priors and involve multi-stage training, thereby limiting their generalization and deployment in shape reasoning and understanding. Driven by the tendency of high-dimensional semantically similar features to lie in or near low-dimensional subspaces, we introduce a one-stage, fully unsupervised framework towards semantic-aware shape representation. This framework produces joint instance segmentation, semantic segmentation, and shape abstraction through sparse representation and feature alignment of object parts in a high-dimensional space. For sparse representation, we devise a sparse latent membership pursuit method that models each object part feature as a sparse convex combination of point features at either the semantic or instance level, promoting part features in the same subspace to exhibit similar semantics. For feature alignment, we customize an attention-based strategy in the feature space to align instance- and semantic-level object part features and reconstruct the input shape using both of them, ensuring geometric reusability and semantic consistency of object parts. To firm up semantic disambiguation, we construct cascade unfrozen learning on geometric parameters of object parts. Experiments conducted on benchmark datasets confirm that our approach results in instance- and semantic-level joint segmentation and shape abstraction with repeatable primitives, providing coherent semantic interpretations of 3D object shapes across categories in a one-stage, fully unsupervised manner, without relying on annotations or heuristic semantic priors. Code will be released at https://github.com/L-Jiaxin/AISSR.

Problem

• Unsupervised joint instance and semantic segmentation for 3D shapes
• Shape abstraction with reusable primitives without annotations
• Aligning sparse semantic-instance features for consistent part representation

Innovation

• Unsupervised sparse representation for shape abstraction
• Attention-based feature alignment strategy
• Cascade unfrozen learning for semantic disambiguation