DrivingAgent: Design and Scheduling Agents for Autonomous Driving Systems

2026-06-10

AI Summary

This work addresses the limitations of autonomous driving systems that rely heavily on manual model design and lack intelligent real-time scheduling mechanisms. To overcome these challenges, the authors propose DrivingAgent, a framework that automates module development during the design phase and introduces a lightweight LLM-driven dynamic orchestration mechanism during inference to meet stringent real-time requirements. The core innovation lies in the first-of-its-kind joint optimization of design and scheduling tasks, integrating hypernetwork training with reinforcement learning. Additionally, a structured memory mechanism is devised, combining long-term memory with timestamp-aware short-term context to enable sustained system operation. Experimental results demonstrate that the proposed approach significantly improves the speed–accuracy trade-off on the nuScenes and Bench2Drive benchmarks.

Abstract

Many autonomous driving systems are increasingly incorporating foundation models to improve generalization and handle long-tail scenarios. However, this trend introduces two key challenges: (i) the manual and labor-intensive process of designing and integrating new models, and (ii) the lack of intelligent, dynamic scheduling mechanisms to meet strict real-time constraints. While Large Language Model (LLM)-based agents offer a promising avenue for automation, existing frameworks are ill-suited for autonomous driving. Specifically, they fail to distinguish between the fundamentally different requirements of system design and real-time scheduling, treat modules as opaque black boxes, and are not designed for continuous operation. To address these limitations, we propose DrivingAgent, a novel agent framework tailored to the dual challenges of autonomous driving system design and scheduling. In the design phase, DrivingAgent automates module development by interpreting system architecture, generating code, and validating modules via super-network training. In the scheduling phase, it employs a lightweight LLM trained with reinforcement learning to dynamically orchestrate system modules in real time, supported by a structured memory that integrates long-term storage with timestamped short-term context. Experimental results demonstrate that DrivingAgent achieves a superior speed--accuracy trade-off on both the nuScenes and Bench2Drive benchmarks.

Problem

• autonomous driving
• foundation models
• real-time scheduling
• system design
• long-tail scenarios

Innovation

• DrivingAgent
• autonomous driving
• LLM-based agent
• dynamic scheduling
• super-network training