A holistic perception system of internal and external monitoring for ground autonomous vehicles: AutoTRUST paradigm

2025-08-25

AI Summary

To address the fragmentation between external and internal perception in ground autonomous vehicles and insufficient intelligence in cabin interaction, this paper proposes a multimodal integrated perception system. Methodologically, it establishes a unified architecture that jointly processes in-cabin perception—including multi-camera facial recognition and thermal comfort analysis—and out-of-cabin perception—including low-cost LiDAR-based semantic segmentation and 3D point cloud super-resolution—while incorporating a large language model (LLM)-driven virtual assistant for adaptive human–vehicle interaction. The key contribution lies in the first integration of lightweight LiDAR semantic segmentation with an LLM-based agent within a vehicular edge computing system, enabling high-accuracy, low-latency cross-domain perception closure. Experimental deployment on a real-world electric vehicle platform demonstrates average improvements of 12.6% in module accuracy, 37% reduction in response latency, and 28.4% increase in user interaction satisfaction.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper introduces a holistic perception system for internal and external monitoring of autonomous vehicles, with the aim of demonstrating a novel AI-leveraged self-adaptive framework of advanced vehicle technologies and solutions that optimize perception and experience on-board. Internal monitoring system relies on a multi-camera setup designed for predicting and identifying driver and occupant behavior through facial recognition, exploiting in addition a large language model as virtual assistant. Moreover, the in-cabin monitoring system includes AI-empowered smart sensors that measure air-quality and perform thermal comfort analysis for efficient on and off-boarding. On the other hand, external monitoring system perceives the surrounding environment of vehicle, through a LiDAR-based cost-efficient semantic segmentation approach, that performs highly accurate and efficient super-resolution on low-quality raw 3D point clouds. The holistic perception framework is developed in the context of EU's Horizon Europe programm AutoTRUST, and has been integrated and deployed on a real electric vehicle provided by ALKE. Experimental validation and evaluation at the integration site of Joint Research Centre at Ispra, Italy, highlights increased performance and efficiency of the modular blocks of the proposed perception architecture.

Problem

• Develops holistic internal-external monitoring for autonomous vehicles
• Uses AI for driver behavior prediction and cabin environment analysis
• Implements LiDAR-based semantic segmentation for environmental perception

Innovation

• Multi-camera driver monitoring with facial recognition
• LiDAR semantic segmentation for environment perception
• AI sensors analyzing cabin air quality comfort