Vanilla ViT for Automotive Point Cloud Semantic Segmentation

2026-05-29

AI Summary

This work proposes VaViT, a novel approach that successfully adapts the original Vision Transformer—characterized by its structural simplicity, absence of convolutional operations, and lack of hierarchical design—to large-scale LiDAR point cloud semantic segmentation for autonomous driving. Addressing the limitations of existing methods that rely on complex U-Net or local attention architectures and struggle to balance efficiency and performance, VaViT introduces a tailored point cloud tokenizer, a lightweight decoder segmentation head, and point-cloud-specific data augmentation strategies. Despite preserving the native simplicity of the vanilla ViT, VaViT achieves state-of-the-art or comparable performance on nuScenes, SemanticKITTI, and the Waymo Open Dataset, thereby demonstrating the effectiveness and efficiency of a pure Transformer architecture for point cloud semantic segmentation.

Abstract

Plain Transformers have become the de-facto architecture for processing text, audio, image, and video, offering a unified backbone for multimodal learning. However, state-of-the-art architectures for point cloud semantic segmentation remain dominated by U-Nets architectures where convolutions are interleaved with local or windowed attentions. In this work, we show how to effectively leverage vanilla, non-hierarchical ViTs for segmentation of large-scale automotive lidar scenes. We bridge the performance gap thanks to a carefully designed tokenizer, a lightweight decoder segmentation head, and tailored data augmentations. Our approach, VaViT for Vanilla ViT, matches or exceeds the performance of state-of-the-art methods while maintaining the simplicity of ViT architecture. We provide extensive evaluations on nuScenes, SemanticKITTI, and Waymo Open Dataset to validate the efficiency of our method. Code and models are available at https://github.com/valeoai/VaViT.

Problem

• Vanilla ViT
• point cloud semantic segmentation
• automotive lidar
• transformer architecture

Innovation

• Vanilla ViT
• point cloud segmentation
• automotive LiDAR
• tokenizer design
• lightweight decoder