Stipple: Real-Time Incremental Gaussian Splatting with Visual-Inertial Tracking

2026-08-01

AI Summary

This work addresses the limitation of traditional 3D Gaussian Splatting (3DGS) in real-time applications, as its reliance on heavy offline preprocessing hinders deployment in robotics and extended reality scenarios requiring live 3D reconstruction. To overcome this, we present the first incremental 3DGS framework capable of real-time operation, which tightly integrates visual-inertial SLAM (VI-SLAM) with 3DGS by directly leveraging tracking poses to drive efficient incremental training. Built upon the Basalt visual-inertial odometry system and Brush—a GPU-agnostic 3DGS engine implemented in Rust—our approach employs a parallelized incremental training architecture with multithreaded optimizations. This design enables fully synchronized camera pose estimation and scene reconstruction while maintaining high-fidelity rendering, entirely eliminating the need for offline preprocessing.

Abstract

3D Gaussian Splatting (3DGS) provides efficient rendering of photo-realistic scenes, but its heavy preprocessing and training steps make it a poor fit for applications that require real-time reconstruction in robotics or XR. This capability is important since it allows immediate feedback and interaction with new environments. Visual-inertial odometry (VIO) and simultaneous localization and mapping (VI-SLAM) systems, on the other hand, specifically target these real-time applications, which makes them a good choice for integration with 3DGS. We propose a new method that tracks and reconstructs simultaneously in real-time by leveraging an efficient visual-inertial tracking system based on Basalt together with a novel incremental method built on top of Brush, an efficient Rust-based GPU-vendor-agnostic implementation of 3D Gaussian Splatting. We show that many of the heavy preprocessing and training steps of 3DGS can be replaced with a more efficient incremental training strategy that has direct access to the information generated by the visual-inertial tracking system. Furthermore, we propose and combine multiple practical improvements to increase the efficiency of the training pipeline and adapt it to run in real-time, parallel to the tracking thread. This work highlights the value of exploiting the complementary nature of SLAM and 3DGS, and how that can lead to promising results for real-time 3D reconstruction.

Problem

• 3D Gaussian Splatting
• real-time reconstruction
• visual-inertial odometry
• SLAM
• incremental training

Innovation

• incremental Gaussian splatting
• visual-inertial tracking
• real-time 3D reconstruction
• SLAM integration
• efficient training pipeline