Automated Vehicle Driver Monitoring Dataset from Real-World Scenarios

2024-08-19

2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC)

AI Summary

To address safety risks in Level 3 (L3) automated driving—specifically, delayed driver takeover responses caused by misjudgment of system capabilities and subsequent engagement in non-driving-related tasks—this paper introduces Real-DAM, the first open-source, real-world driver activity monitoring dataset. Real-DAM is collected using in-vehicle multimodal sensors under diverse lighting conditions, weather scenarios, and representative secondary-task contexts, featuring fine-grained annotations, large scale, and high variability. Unlike existing synthetic datasets, Real-DAM significantly mitigates the simulation-to-reality domain shift, bridging a critical gap in real-vehicle generalization data for training. Experimental results demonstrate that deep learning models trained on Real-DAM achieve substantially improved accuracy and robustness in driver activity recognition under real-world conditions. As such, Real-DAM establishes a new benchmark for human–machine collaborative safety warning research in L3 automation.

Technology Category

• Application Domains: Mobility, Driving & Flight
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Computer Vision: 3D Computer Vision

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements

Abstract

From SAE Level 3 of automation onwards, drivers are allowed to engage in activities that are not directly related to driving during their travel. However, in level 3, a misunderstanding of the capabilities of the system might lead drivers to engage in secondary tasks, which could impair their ability to react to challenaing traffic situations. Anticipating driver activity allows for early detection of risky behaviors, to prevent accidents. To be able to predict the driver activity, a Deep Learning network needs to be trained on a dataset. However, the use of datasets based on simulation for training and the migration to real-world data for prediction has proven to be suboptimal. Hence, this paper presents a real-world driver activity dataset, openly accessible on IEEE Dataport, which encompasses various activities that occur in autonomous driving scenarios under various illumination and weather conditions. Results from the training process showed that the dataset provides an excellent benchmark for implementing models for driver activity recognition.

Problem

• Dataset for real-world driver activity monitoring in autonomous vehicles
• Addressing limitations of simulation-based training for driver behavior prediction
• Enhancing safety by detecting risky behaviors in SAE Level 3 automation

Innovation

• Real-world driver activity dataset creation
• Deep Learning for activity recognition
• Various illumination and weather conditions coverage