LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture

2025-05-08

AI Summary

This work addresses key limitations—poor task scalability, insufficient safety guarantees, and non-smooth task transitions—arising from the decoupled integration of Large Vision-Language Models (LVLMs) and Model Predictive Control (MPC) in autonomous driving. We propose a closed-loop LVLM-MPC coupling architecture. Our method introduces: (i) an LVLM-driven symbolic task parsing mechanism that generates high-level instructions in real time and triggers an MPC Builder to automatically synthesize safety-constrained, task-specific controllers; and (ii) a task feasibility feedback and rejection mechanism to ensure instruction executability. The system unifies vision-language understanding, symbolic reasoning, automatic controller synthesis, and safety-aware optimization. Evaluated in highway simulation, it enables seamless switching among complex driving tasks, achieving a 32% improvement in task success rate and zero collisions—effectively bridging LVLMs’ semantic flexibility with MPC’s motion-level reliability.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This paper proposes a novel Large Vision-Language Model (LVLM) and Model Predictive Control (MPC) integration framework that delivers both task scalability and safety for Autonomous Driving (AD). LVLMs excel at high-level task planning across diverse driving scenarios. However, since these foundation models are not specifically designed for driving and their reasoning is not consistent with the feasibility of low-level motion planning, concerns remain regarding safety and smooth task switching. This paper integrates LVLMs with MPC Builder, which automatically generates MPCs on demand, based on symbolic task commands generated by the LVLM, while ensuring optimality and safety. The generated MPCs can strongly assist the execution or rejection of LVLM-driven task switching by providing feedback on the feasibility of the given tasks and generating task-switching-aware MPCs. Our approach provides a safe, flexible, and adaptable control framework, bridging the gap between cutting-edge foundation models and reliable vehicle operation. We demonstrate the effectiveness of our approach through a simulation experiment, showing that our system can safely and effectively handle highway driving while maintaining the flexibility and adaptability of LVLMs.

Problem

• Ensuring safety and smooth task switching in autonomous driving
• Integrating LVLM high-level planning with low-level MPC feasibility
• Bridging foundation models and reliable vehicle operation safely

Innovation

• LVLM and MPC integration for autonomous driving
• Automatic MPC generation based on LVLM commands
• Safety-aware task switching with feasibility feedback