Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection

2026-06-01

AI Summary

Self-supervised driver distraction detection in multimodal video is highly susceptible to viewpoint variations, occlusions, and semantic overlap, which compromise the reliability of positive and negative samples. To address this challenge, this work proposes a multimodal global alignment framework that leverages cycle consistency to generate soft targets, thereby relaxing the rigid hard-negative assumption. Additionally, a similarity-distribution-based weighting mechanism is introduced to suppress interference from noisy positive samples. This approach extends conventional pairwise alignment to a global multimodal alignment paradigm. Evaluated on the Drive&Act dataset encompassing RGB, infrared, depth, and skeleton modalities, the proposed method significantly outperforms existing approaches and demonstrates strong generalization capability in cross-view scenarios.

Abstract

Robust self-supervised learning of multi-modal video representations is critical for real-world applications such as driver distraction detection, where multiple sensors provide complementary but noisy signals. Conventional contrastive objectives, such as InfoNCE, assume all negatives are equally informative and all positives are reliable. However, this assumption is frequently violated in multi-modal data due to viewpoint changes, occlusions, or semantic overlap across modalities. In this work, we propose a novel framework for multi-modal global alignment that addresses these challenges by jointly modeling faulty negatives and unreliable or faulty positives. We introduce soft targets derived from cycle-consistency scores to relax the hard-negative assumption, and a weighting mechanism based on similarity distributions to mitigate the impact of noisy or faulty positives. Our approach extends traditional pairwise alignment to a principled global multi-modal setting, aggregating alignment information across all modality pairs. We evaluate our method on the Drive&Act dataset, demonstrating that it consistently outperforms both pairwise and existing global alignment baselines across RGB, IR, Depth, and Skeleton modalities. Cross-view ablation studies further show strong generalization to unseen camera perspectives, highlighting the robustness of our representations. Overall, our framework provides a scalable and effective solution for self-supervised global multi-modal representation learning, enabling reliable driver distraction detection and pioneering in real-world multi-modal video understanding. Our code will be published on GitHub.

Problem

• multi-modal video
• self-supervised learning
• driver distraction detection
• noisy labels
• representation alignment

Innovation

• multi-modal alignment
• self-supervised learning
• global representation learning
• cycle-consistency
• driver distraction detection