LLM-Generated Fault Scenarios for Evaluating Perception-Driven Lane Following in Autonomous Edge Systems

2026-04-01

AI Summary

This work addresses the challenge of comprehensive safety testing for resource-constrained edge-based autonomous driving vision systems, which struggle to perform real-time evaluation under diverse failure conditions. Existing approaches rely on static datasets or manual fault injection, resulting in insufficient coverage. To overcome this limitation, the authors propose a decoupled offline-online fault injection framework: in the offline phase, large language models generate semantically meaningful failure scenarios, and latent diffusion models synthesize high-fidelity sensor degradation data; in the online phase, lightweight, real-time fault-aware inference is enabled via precomputed lookup tables. This approach uniquely integrates large language models and diffusion models to automatically produce structured, high-fidelity perception faults. Evaluation across 460 scenarios shows that while the system achieves an R² of 0.85 on clean data, generated faults such as fog can increase RMSE by 99% and reduce localization accuracy (error < 0.10) to 31.0%, exposing critical gaps in conventional evaluation protocols for edge deployment.

Abstract

Deploying autonomous vision systems on edge devices faces a critical challenge: resource constraints prevent real-time and predictable execution of comprehensive safety tests. Existing validation methods depend on static datasets or manual fault injection, failing to capture the diverse environmental hazards encountered in real-world deployment. To address this, we introduce a decoupled offline-online fault injection framework. This architecture separates the validation process into two distinct phases: a computationally intensive Offline Phase and a lightweight Online Phase. In the offline phase, we employ Large Language Models (LLMs) to semantically generate structured fault scenarios and Latent Diffusion Models (LDMs) to synthesize high-fidelity sensor degradations. These complex fault dynamics are distilled into a pre-computed lookup table, enabling the edge device to perform real-time fault-aware inference without running heavy AI models locally. We extensively validated this framework on a ResNet18 lane-following model across 460 fault scenarios. Results show that while the model achieves a baseline R^2 of approximately 0.85 on clean data, our generated faults expose significant robustness degradation, with RMSE increasing by up to 99% and within-0.10 localization accuracy dropping to as low as 31.0% under fog conditions, demonstrating the inadequacy of normal-data evaluation for real-world edge AI deployment.

Problem

• autonomous edge systems
• fault injection
• perception-driven lane following
• resource constraints
• safety validation

Innovation

• LLM-generated fault scenarios
• latent diffusion models
• edge AI robustness
• decoupled offline-online validation
• perception-driven lane following