Not All Uncertainty Matters: Simulation-in-the-Loop Fast-Slow Reasoning for Decision-Critical Autonomous Driving System

2026-10-07

AI Summary

This study addresses the latency conflict between cloud-based large models and real-time decision-making in autonomous driving, as well as the lack of uncertainty-aware planning benefit evaluation in existing methods. We propose SIGMA, a framework that pioneers embedding the planner into the uncertainty assessment pipeline. It introduces the Expected Planning Gain (EPG) metric to quantify the decision value of resolving uncertainty. By integrating vision-language models, simulation-in-the-loop mechanisms, and geometric-semantic uncertainty modeling, SIGMA enables task-oriented fast-slow collaborative reasoning under resource constraints. Experiments on CARLA demonstrate that the proposed method reduces ineffective cloud interactions by 50%, improves navigation success rate by over 6%, and decreases completion time in dynamic scenarios by 26.2%.

Abstract

Large vision-language models (VLMs) provide powerful open-world perception and reasoning for autonomous driving, but their high computational cost and inference latency make continuous cloud-side use impractical. This motivates fast--slow collaboration, where efficient onboard modules handle real-time perception and control while cloud models provide high-level reasoning only when needed. The key challenge is deciding when cloud reasoning should influence time-critical driving decisions. Existing methods often rely on perception uncertainty, heuristic triggers, or resource-driven policies, without assessing whether resolving an uncertainty will improve planning. We propose \textbf{SIGMA}, a simulation-in-the-loop framework for task-oriented fast--slow collaboration. SIGMA embeds the planner into uncertainty assessment and evaluates how plausible scene realizations under semantic and geometric uncertainty affect feasible trajectories and planning cost. Based on these outcomes, it estimates the expected reduction in planning cost from resolving uncertainty. We further introduce expected planning gain (EPG), a decision-level metric for cloud invocation, cloud-guidance integration, and request prioritization under deadline and resource constraints. Experiments in CARLA show that SIGMA reduces unnecessary cloud interactions while improving planning, efficiency, and navigation success in static and dynamic obstacle scenarios. Compared with fixed-period collaboration, SIGMA reduces unnecessary cloud interactions by 50\%, improves navigation success by more than 6\%, and cuts finish time by up to 26.2\% in dynamic scenarios.

Problem

• Autonomous Driving
• Vision-Language Models
• Fast-Slow Collaboration
• Uncertainty Assessment
• Decision-Critical Planning

Innovation

• Simulation-in-the-Loop
• Fast-Slow Reasoning
• Expected Planning Gain
• Vision-Language Models
• Autonomous Driving