MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios

2024-12-27

AI Summary

To address the joint modeling challenge of semantic understanding and risk-object localization from a single forward-facing image in autonomous driving, this paper proposes DRAMA: a dual-resolution vision encoder enhances multi-scale risk perception; LLaMA is fine-tuned to generate driving scene descriptions, identify risky behaviors, and recommend ego-vehicle actions; and, for the first time, regression tokens are embedded within a Transformer decoder to enable end-to-end co-optimization of language generation and spatial localization. DRAMA is the first framework enabling semantic–localization joint modeling from a single image input. On the DRAMA-ROLISP/SRIS benchmark, it achieves 80.1% BLEU-1 and 298.5% CIDEr for scene understanding, and 59.6% accuracy for risk-object localization—surpassing all existing image- and video-based methods.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multimodal large language models (MLLMs) have shown satisfactory effects in many autonomous driving tasks. In this paper, MLLMs are utilized to solve joint semantic scene understanding and risk localization tasks, while only relying on front-view images. In the proposed MLLM-SUL framework, a dual-branch visual encoder is first designed to extract features from two resolutions, and rich visual information is conducive to the language model describing risk objects of different sizes accurately. Then for the language generation, LLaMA model is fine-tuned to predict scene descriptions, containing the type of driving scenario, actions of risk objects, and driving intentions and suggestions of ego-vehicle. Ultimately, a transformer-based network incorporating a regression token is trained to locate the risk objects. Extensive experiments on the existing DRAMA-ROLISP dataset and the extended DRAMA-SRIS dataset demonstrate that our method is efficient, surpassing many state-of-the-art image-based and video-based methods. Specifically, our method achieves 80.1% BLEU-1 score and 298.5% CIDEr score in the scene understanding task, and 59.6% accuracy in the localization task. Codes and datasets are available at https://github.com/fjq-tongji/MLLM-SUL.

Problem

• Autonomous Driving
• Scene Understanding
• Safety Enhancement

Innovation

• MLLM-SUL
• Dual-path Image Processing
• Transformers for Scene Understanding