SurgGMF: Fully Causal Gaussian Motion Forecasting for Anticipatory Surgical Scene Rendering

2026-09-28

AI Summary

This study addresses the limitation of existing neural rendering methods in surgical scenes, which are restricted to reconstruction rather than predicting future states. To overcome this, we propose a fully causal Gaussian motion prediction framework that leverages temporal models—including TKAN, GRU, and LSTM—to forecast displacement, scale, and rotation residuals of historical Gaussian fields. A fully causal rendering protocol is designed to strictly prevent target information leakage, advancing 3D Gaussian representations from retrospective reconstruction to predictive modeling. Experiments on twelve video clips demonstrate that learning-based approaches significantly outperform classical dynamics baselines. Specifically, TKAN achieves the highest accuracy, while GRU and LSTM offer lower inference latency, validating the effectiveness of the proposed framework for prospective rendering in surgical scenarios.

Abstract

Dynamic surgical scene modeling is essential for robotic perception, simulation, and decision support. Although existing neural rendering methods enable efficient reconstruction and rendering of deformable surgical scenes, they remain primarily focused on observed-frame reconstruction rather than forecasting future scene states. To this end, we present SurgGMF, a fully causal Gaussian motion forecasting framework for anticipatory surgical scene rendering. Rather than predicting future RGB images directly, SurgGMF forecasts future Gaussian motion states represented by position, scale, and rotation residuals (X/S/R) from historical Gaussian motion fields. To prevent target leakage, we introduce a full-causal-last rendering protocol, where future Gaussian states are rendered without accessing target-frame Gaussian attributes while preserving causal appearance propagation. We evaluate SurgGMF on 12 EndoNeRF and StereoMIS video slices using neural temporal learners and classical dynamics baselines under a unified forecasting protocol. Learned Gaussian motion forecasting consistently outperforms classical dynamics baselines in render space, demonstrating gains beyond hand-crafted state extrapolation. Latency analysis further reveals an accuracy--efficiency trade-off: under the current implementations, TKAN achieves the highest accuracy, whereas GRU and LSTM provide more favorable module-level latency profiles. These results establish SurgGMF as a reproducible framework for causal Gaussian motion forecasting and advance surgical Gaussian representations from retrospective reconstruction toward predictive scene modeling.

Problem

• surgical scene forecasting
• dynamic scene modeling
• Gaussian motion forecasting
• anticipatory rendering

Innovation

• Gaussian Motion Forecasting
• Causal Rendering
• Surgical Scene Modeling
• Neural Rendering
• Anticipatory Rendering