Sensitivity Analysis of GRU, LSTM and Transformer Encoder in Classification of Automated Driving Systems

2026-07-24

AI Summary

This study addresses the challenge of accurately identifying the currently active Level 2 advanced driver assistance system—such as Tesla Autopilot, Comma Openpilot, or Cadillac Super Cruise—or manual driving mode using only vehicle telemetry data, while systematically evaluating model robustness under data degradation. The work proposes the first multidimensional telemetry degradation simulation framework tailored to this task, encompassing five realistic perturbation types: additive noise, cumulative drift, cross-channel correlated noise, temporal jitter, and event-level disturbances. Experimental results demonstrate that GRU, LSTM, and Transformer encoder models achieve strong performance on clean data (macro-F1 scores of 0.90–0.93) and remain robust to event-level perturbations (F1 ≥ 0.87). However, temporal jitter severely degrades performance (F1 dropping to 0.44–0.50), highlighting a critical vulnerability in temporal modeling for this recognition task.

Abstract

Automated driving systems (ADSs) are becoming ubiquitous. Future Software Defined Vehicles (SDVs) may be able to run multiple ADSs, both native and aftermarket such as Comma.ai's Openpilot. Monitoring systems to independently verify which automated driving system is active are important for safety monitoring, regulatory compliance, insurance assessment, and anomaly detection. In this paper, we first evaluate the effectiveness of three sequence-based classification models: Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM) networks, and a Transformer encoder model for identifying Level 2 automated driving systems using vehicle telematics data alone: Comma Openpilot, Tesla Autopilot, and Cadillac Super Cruise, along with manual driving. All three models achieve strong clean-data performance with macro F1-scores of 0.92 (GRU), 0.90 (LSTM), and 0.93 (Transformer encoder model) when trained on clean data; threat-matched training yields 0.904-0.916 macro F1 with only a modest clean-data penalty. Second, we introduce a modular robustness evaluation framework that simulates realistic telematics degradation through five corruption families at five severity levels (L1-L5). Continuous channels are perturbed using additive white Gaussian noise with cumulative drift, correlated cross-channel noise, and temporal jitter. Binary event signals are subjected to burst loss, delayed transitions, spurious toggles and cross-feature inconsistencies inspired by communication errors. Robustness is measured using macro-F1, which gives equal weight to each class and is suitable for imbalanced multiclass evaluation. Our evaluation reveals a sharp failure-mode split: event-level corruptions reduce macro-F1 only slightly (greater than equal to 0.87 at L5), while temporal jitter collapses macro-F1 to 0.44-0.50 across GRU, LSTM, and Transformer encoder model.

Problem

• Automated Driving Systems
• Telematics Data
• Classification Robustness
• Sequence-based Models
• Data Degradation

Innovation

• robustness evaluation framework
• temporal jitter
• telematics degradation
• sequence-based classification
• automated driving system identification