Event-Driven Dynamic Scene Depth Completion

2025-05-19

AI Summary

To address severe degradation in RGB-D depth completion under dynamic scenes—caused by concurrent ego-motion and object motion—this paper introduces EventDC, the first event-camera-based depth completion framework. Methodologically, it proposes two novel modules: Event-Modulated Alignment (EMA) for motion-aware pixel-level feature redistribution, and Local Depth Filtering (LDF) for mask-guided depth refinement. The architecture incorporates event-stream encoding, motion-conditioned deformable convolution, and multi-scale feature fusion, optimized via a global-local joint loss. Contributions include: (i) the first event-driven depth completion benchmark, comprising both real-world and synthetic dynamic-scene data; and (ii) the open-sourcing of the first end-to-end event-based depth completion model. Experiments demonstrate that EventDC reduces depth error at dynamic edges by 23.6%, significantly enhancing robustness and accuracy in fast-motion scenarios.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Depth completion in dynamic scenes poses significant challenges due to rapid ego-motion and object motion, which can severely degrade the quality of input modalities such as RGB images and LiDAR measurements. Conventional RGB-D sensors often struggle to align precisely and capture reliable depth under such conditions. In contrast, event cameras with their high temporal resolution and sensitivity to motion at the pixel level provide complementary cues that are %particularly beneficial in dynamic environments.To this end, we propose EventDC, the first event-driven depth completion framework. It consists of two key components: Event-Modulated Alignment (EMA) and Local Depth Filtering (LDF). Both modules adaptively learn the two fundamental components of convolution operations: offsets and weights conditioned on motion-sensitive event streams. In the encoder, EMA leverages events to modulate the sampling positions of RGB-D features to achieve pixel redistribution for improved alignment and fusion. In the decoder, LDF refines depth estimations around moving objects by learning motion-aware masks from events. Additionally, EventDC incorporates two loss terms to further benefit global alignment and enhance local depth recovery. Moreover, we establish the first benchmark for event-based depth completion comprising one real-world and two synthetic datasets to facilitate future research. Extensive experiments on this benchmark demonstrate the superiority of our EventDC.

Problem

• Addresses depth completion challenges in dynamic scenes with rapid motion
• Proposes event-driven framework to improve RGB-D alignment using event cameras
• Introduces benchmark for event-based depth completion with real and synthetic data

Innovation

• Event-driven depth completion framework
• Event-Modulated Alignment for RGB-D fusion
• Local Depth Filtering for motion-aware refinement