D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models

2026-07-21

AI Summary

Current multimodal large language models for autonomous driving primarily rely on 2D visual data and struggle to effectively integrate sparse, unstructured 3D LiDAR point clouds with temporal camera sequences, limiting comprehensive understanding of complex traffic scenarios. This work proposes D3VL, a novel framework that, for the first time, enables end-to-end fusion of 3D temporal LiDAR point clouds and video data within a unified architecture. It introduces lightweight mechanisms specifically designed to address the challenges posed by LiDAR sparsity and non-grid structure. The approach supports language-model-based scene understanding and safety-oriented question answering, achieving an 11% improvement over baseline methods on the KITTI QA benchmark. Furthermore, the authors release an extended Waymo QA dataset, demonstrating the model’s effectiveness in handling 3D temporal information across diverse driving conditions.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have triggered the development of end-to-end MLLMs for autonomous driving. However, the main emphasis to date has been for MLLMs using 2D images and videos. In contrast, this paper considers MLLM effectiveness using 3D sensors, particularly LiDAR and stereo cameras. LiDAR presents unique challenges to integration within an MLLM, largely because of data sparsity and lack of a grid structure for the data. For similar reasons, fusion of camera and LiDAR data within an MLLM pipeline is also uncommon. However, most autonomous systems rely on LiDAR-based sensing, and incorporating 3D data has been proven to improve performance in traditional 3D scene perception tasks. This paper presents D3VL, a novel MLLM framework that integrates 2D and 3D time-series data in a single but simple architecture. The model aims to answer questions involving traffic scene understanding and safety. D3VL shows an 11% improvement in the KITTI Question-Answering (QA) dataset compared to baseline methods in processing 2D and 3D time-series data. This paper further introduces the Waymo QA dataset extension, which assesses models' capabilities in processing 3D and time-series data under diverse driving conditions. D3VL implementation code and WaymoQA extension can be found on our supplemental website: https://automotivesafety-lvlm.github.io

Problem

• 3D LiDAR
• Multimodal Large Language Models
• Autonomous Driving
• Time Series Data
• Scene Understanding

Innovation

• Multimodal Large Language Models
• 3D Time Series
• LiDAR-Camera Fusion
• Autonomous Driving
• Scene Understanding