S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Driving

2024-10-30

arXiv.org

AI Summary

To address self-supervised representation bias in autonomous driving—arising from imbalanced object category and scale distributions and complex scene geometry—this paper proposes a semantic-structural co-guided clustering framework. Our method introduces three novel mechanisms: (1) semantic-distribution-consistent clustering to mitigate long-tail bias; (2) object-diversity-consistent spatial clustering to enhance representation learning for rare classes; and (3) depth-guided spatial clustering to improve geometric awareness. The approach jointly integrates semantic consistency constraints, multi-scale spatial clustering, depth-map-based geometric regularization, and contrastive self-supervised learning. Extensive experiments on nuScenes, nuImages, and Cityscapes demonstrate significant improvements in downstream semantic segmentation and 3D object detection performance. Moreover, the framework exhibits strong cross-domain generalization capability, validating its robustness across diverse urban driving scenarios.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Recent self-supervised clustering-based pre-training techniques like DINO and Cribo have shown impressive results for downstream detection and segmentation tasks. However, real-world applications such as autonomous driving face challenges with imbalanced object class and size distributions and complex scene geometries. In this paper, we propose S3PT a novel scene semantics and structure guided clustering to provide more scene-consistent objectives for self-supervised training. Specifically, our contributions are threefold: First, we incorporate semantic distribution consistent clustering to encourage better representation of rare classes such as motorcycles or animals. Second, we introduce object diversity consistent spatial clustering, to handle imbalanced and diverse object sizes, ranging from large background areas to small objects such as pedestrians and traffic signs. Third, we propose a depth-guided spatial clustering to regularize learning based on geometric information of the scene, thus further refining region separation on the feature level. Our learned representations significantly improve performance in downstream semantic segmentation and 3D object detection tasks on the nuScenes, nuImages, and Cityscapes datasets and show promising domain translation properties.

Problem

• Autonomous Driving
• Object Recognition
• Adaptive Learning

Innovation

• Scene-aware Object Recognition
• Structural Information Utilization
• Depth-aware 3D Object Identification