DVS-PedX: Synthetic-and-Real Event-Based Pedestrian Dataset

2025-09-04

AI Summary

Pedestrian detection and crossing-intent analysis under adverse weather conditions remain challenging for conventional vision systems. Method: We introduce the first cross-domain event dataset tailored for Spiking Neural Networks (SNNs), integrating synthetic event streams from CARLA simulations with real-world JAAD driving videos converted to AEDAT format via v2e. The dataset spans diverse lighting and weather conditions and provides synchronized event frames, RGB frames, and fine-grained frame-level behavioral annotations, enabling SNN training and evaluation on SpikingJelly. Contribution/Results: Experiments validate the efficacy of SNN baselines and—crucially—quantify, for the first time, a substantial “simulation-to-reality” domain gap. This dataset constitutes the first large-scale, multimodal, intent-annotated benchmark for neuromorphic vision in intelligent transportation perception, advancing research in domain adaptation and multimodal fusion.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Event cameras like Dynamic Vision Sensors (DVS) report micro-timed brightness changes instead of full frames, offering low latency, high dynamic range, and motion robustness. DVS-PedX (Dynamic Vision Sensor Pedestrian eXploration) is a neuromorphic dataset designed for pedestrian detection and crossing-intention analysis in normal and adverse weather conditions across two complementary sources: (1) synthetic event streams generated in the CARLA simulator for controlled "approach-cross" scenes under varied weather and lighting; and (2) real-world JAAD dash-cam videos converted to event streams using the v2e tool, preserving natural behaviors and backgrounds. Each sequence includes paired RGB frames, per-frame DVS "event frames" (33 ms accumulations), and frame-level labels (crossing vs. not crossing). We also provide raw AEDAT 2.0/AEDAT 4.0 event files and AVI DVS video files and metadata for flexible re-processing. Baseline spiking neural networks (SNNs) using SpikingJelly illustrate dataset usability and reveal a sim-to-real gap, motivating domain adaptation and multimodal fusion. DVS-PedX aims to accelerate research in event-based pedestrian safety, intention prediction, and neuromorphic perception.

Problem

• Event-based pedestrian detection and crossing-intention analysis
• Dataset for normal and adverse weather conditions
• Addressing sim-to-real gap in neuromorphic perception

Innovation

• Synthetic and real event-based dataset creation
• Spiking neural networks for pedestrian detection
• Domain adaptation and multimodal fusion techniques