BeSplat -- Gaussian Splatting from a Single Blurry Image and Event Stream

2024-12-26

AI Summary

This work addresses the ill-posed problem of novel-view synthesis from a single motion-blurred image. We propose the first end-to-end reconstruction framework that jointly leverages event streams and 3D Gaussian splatting. To resolve ambiguity induced by single-frame blur, our method co-optimizes an implicit scene representation (3D Gaussian splatting) and the camera motion trajectory (parameterized as a Bézier curve in SE(3)). A physics-informed event stream model is integrated, alongside a multi-objective loss combining blurred-image reconstruction and event consistency constraints. To the best of our knowledge, this is the first approach to enable motion-scene co-recovery from a single blurred frame augmented with events within the Gaussian splatting paradigm. Experiments on both synthetic and real-world data demonstrate that our method reconstructs sharp, geometrically consistent novel views and accurately estimates camera trajectories—significantly outperforming existing single-frame motion-deblurring methods.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Novel view synthesis has been greatly enhanced by the development of radiance field methods. The introduction of 3D Gaussian Splatting (3DGS) has effectively addressed key challenges, such as long training times and slow rendering speeds, typically associated with Neural Radiance Fields (NeRF), while maintaining high-quality reconstructions. In this work (BeSplat), we demonstrate the recovery of sharp radiance field (Gaussian splats) from a single motion-blurred image and its corresponding event stream. Our method jointly learns the scene representation via Gaussian Splatting and recovers the camera motion through Bezier SE(3) formulation effectively, minimizing discrepancies between synthesized and real-world measurements of both blurry image and corresponding event stream. We evaluate our approach on both synthetic and real datasets, showcasing its ability to render view-consistent, sharp images from the learned radiance field and the estimated camera trajectory. To the best of our knowledge, ours is the first work to address this highly challenging ill-posed problem in a Gaussian Splatting framework with the effective incorporation of temporal information captured using the event stream.

Problem

• Gaussian splatter extraction
• image quality preservation
• time efficiency

Innovation

• Gaussian Splats
• Temporal Variations
• Angle Matching and Image Clarity