Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models

2026-05-20

AI Summary

This work addresses the challenges of error-prone and inefficient manual 3D vehicle annotation in complex autonomous driving scenarios, particularly under occlusion. To overcome these limitations, the study introduces vision-language models (VLMs) into the 3D annotation pipeline for the first time, leveraging zero-shot inference to predict vehicle make, model, and generation from cropped image regions and generate accurate initial 3D bounding box dimensions. By integrating iterative prompt engineering with Vehicle Make and Model Recognition (VMMR) techniques, the proposed method demonstrates strong generalization across both public and proprietary datasets. It significantly outperforms conventional LiDAR-assisted annotation approaches, markedly improving annotation accuracy while substantially reducing human effort, especially in mitigating annotation failures caused by occlusion.

Abstract

We present an approach to improve 3D vehicle labeling in self-driving applications through zero-shot inference of vehicle information, leveraging Vehicle Make and Model Recognition (VMMR) methods. The proposed approach utilizes a Vision Language Model (VLM) to both infer a vehicle's make, model, and generation from image crops, and output accurate 3D bounding box dimensions to seed manual labeling. We evaluate the impact of iterative prompt engineering and the choice of different VLMs on both vehicle bounding box inference and make/model/generation recognition. When compared to strong baselines, the proposed approach not only shows high accuracy, but also excels in mitigating specific failure modes where VLMs provide better dimensions than initial lidar-aided human annotated labels (e.g., in cases of significant vehicle occlusion). Experiments on both public and proprietary data strongly suggest that our conclusions are generalizable across different labelers and datasets. The results demonstrate that integrating VLMs into the labeling process can reduce manual labeling time while increasing label quality.

Problem

• 3D labeling
• self-driving
• vehicle annotation
• occlusion
• label quality

Innovation

• Vision Language Models
• 3D Vehicle Labeling
• Zero-shot Inference
• Vehicle Make and Model Recognition
• Autonomous Driving