BAT: Learning Event-based Optical Flow with Bidirectional Adaptive Temporal Correlation

2025-03-05

AI Summary

Event cameras produce sparse, asynchronous data, leading to low accuracy and poor density in optical flow estimation. To address this, we propose a bidirectional adaptive temporal correlation modeling framework. Our key contributions are: (1) bidirectional temporal correlation modeling over event voxels; (2) a learnable adaptive temporal sampling mechanism that dynamically selects informative time intervals; and (3) a spatially adaptive motion feature aggregation module enabling future flow prediction from historical events alone. The method is trained end-to-end and achieves first place on the DSEC-Flow benchmark, significantly outperforming state-of-the-art approaches. Qualitatively, the estimated flow fields exhibit sharp object boundaries and rich fine-grained motion details. Quantitatively, our future flow prediction surpasses the E-RAFT warm-start baseline, demonstrating superior generalization and temporal reasoning capability.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Time-Series/Data Streams
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Event cameras deliver visual information characterized by a high dynamic range and high temporal resolution, offering significant advantages in estimating optical flow for complex lighting conditions and fast-moving objects. Current advanced optical flow methods for event cameras largely adopt established image-based frameworks. However, the spatial sparsity of event data limits their performance. In this paper, we present BAT, an innovative framework that estimates event-based optical flow using bidirectional adaptive temporal correlation. BAT includes three novel designs: 1) a bidirectional temporal correlation that transforms bidirectional temporally dense motion cues into spatially dense ones, enabling accurate and spatially dense optical flow estimation; 2) an adaptive temporal sampling strategy for maintaining temporal consistency in correlation; 3) spatially adaptive temporal motion aggregation to efficiently and adaptively aggregate consistent target motion features into adjacent motion features while suppressing inconsistent ones. Our results rank $1^{st}$ on the DSEC-Flow benchmark, outperforming existing state-of-the-art methods by a large margin while also exhibiting sharp edges and high-quality details. Notably, our BAT can accurately predict future optical flow using only past events, significantly outperforming E-RAFT's warm-start approach. Code: extcolor{magenta}{https://github.com/gangweiX/BAT}.

Problem

• Estimates optical flow for event cameras with high dynamic range.
• Overcomes spatial sparsity limitations in event-based optical flow estimation.
• Introduces bidirectional adaptive temporal correlation for accurate motion prediction.

Innovation

• Bidirectional temporal correlation for dense optical flow
• Adaptive temporal sampling for consistency
• Spatially adaptive motion aggregation for efficiency