Leveraging Multimodal-LLMs Assisted by Instance Segmentation for Intelligent Traffic Monitoring

2025-02-16

AI Summary

To address low accuracy, poor interpretability, and insufficient real-time interactivity in dynamic scene understanding for intelligent traffic monitoring, this paper proposes a synergistic architecture integrating the vision-grounded multimodal large language model LLaVA with embedded instance segmentation (Mask R-CNN). Deployed on the Quanser real-time simulation platform, the system fuses multi-view video perception with a natural language query interface to enable semantic-level dynamic analysis and adaptive response to intersection states, congestion evolution, and collision events. Its key innovation lies in the first joint application of vision-grounded multimodal LLMs and lightweight instance segmentation for fine-grained traffic object localization and intent inference, significantly enhancing interpretability and semantic reasoning capability for critical targets. Experimental results demonstrate vehicle position identification accuracy of 84.3% and turning-direction classification accuracy of 76.4%, both surpassing conventional vision-only methods.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

A robust and efficient traffic monitoring system is essential for smart cities and Intelligent Transportation Systems (ITS), using sensors and cameras to track vehicle movements, optimize traffic flow, reduce congestion, enhance road safety, and enable real-time adaptive traffic control. Traffic monitoring models must comprehensively understand dynamic urban conditions and provide an intuitive user interface for effective management. This research leverages the LLaVA visual grounding multimodal large language model (LLM) for traffic monitoring tasks on the real-time Quanser Interactive Lab simulation platform, covering scenarios like intersections, congestion, and collisions. Cameras placed at multiple urban locations collect real-time images from the simulation, which are fed into the LLaVA model with queries for analysis. An instance segmentation model integrated into the cameras highlights key elements such as vehicles and pedestrians, enhancing training and throughput. The system achieves 84.3% accuracy in recognizing vehicle locations and 76.4% in determining steering direction, outperforming traditional models.

Problem

• Develops intelligent traffic monitoring using multimodal-LLMs
• Enhances accuracy in vehicle location and direction detection
• Integrates instance segmentation for improved traffic analysis

Innovation

• Multimodal-LLMs for traffic analysis
• Instance segmentation enhances object detection
• Real-time simulation for traffic monitoring