Towards Trustworthy and Explainable AI for Perception Models: From Concept to Prototype Vehicle Deployment

2026-05-15

AI Summary

This work addresses the critical limitations of current autonomous driving perception models—namely, their lack of interpretability, reliable uncertainty estimation, and robustness—which hinder compliance with safety and regulatory requirements for trustworthy AI. The paper introduces the first 3D perception framework that unifies faithful explainability, calibrated uncertainty, and robustness within a Transformer-based detector. At inference time, the model leverages attention mechanisms to generate faithful saliency maps, complemented by perturbation consistency validation, uncertainty calibration, and robust training strategies. Implemented in an end-to-end prototype vehicle system equipped with an XAI-enabled interactive visualization interface, the proposed approach demonstrates real-time trustworthy perception, significantly enhancing model transparency, robustness, and uncertainty reliability while maintaining high performance.

Abstract

Deep Neural Networks have become the dominant solution for Autonomous Driving perception, but their opacity conflicts with emerging Trustworthy AI guidelines and complicates safety assurance, debugging, and human oversight. While theoretical frameworks for safe and Explainable AI (XAI) exist, concrete implementations of Trustworthy AI for 3D scene understanding remain scarce. We address this gap by proposing a Trustworthy AI perception module that is remarkably robust, integrates faithful explainability, and calibrated uncertainty estimates. Building on a transformer-based detector, we derive explanation from the attention mechanism at inference time and validate their faithfulness using perturbation-based consistency tests. We further integrate an uncertainty estimation and calibration module, and apply robustness-enhancing training methods. Experiments show faithful saliency behavior, improved robustness, and well-calibrated uncertainty estimates. Finally, we deploy these Trustworthy AI elements in a prototype vehicle and provide an XAI Interface that visualizes documentation artifacts, model uncertainty state, and saliency maps, demonstrating the feasibility of trustworthy perception monitoring in real time. Supplementary materials are available at https://tillbeemelmanns.github.io/trustworthy_ai/ .

Problem

• Trustworthy AI
• Explainable AI
• Autonomous Driving
• Perception Models
• 3D Scene Understanding

Innovation

• Trustworthy AI
• Explainable AI (XAI)
• Uncertainty Calibration
• Transformer-based Perception
• Autonomous Driving