Super-Gaussian: Interactive Scene Editing for 3D Gaussian Splatting and NLI-Based Volume Visualization in Virtual Reality

2026-08-05

AI Summary

This work addresses the challenges of volumetric visualization in virtual reality, including high rendering costs, user fatigue, and the inefficiency of existing 3D Gaussian editing methods that rely on offline segmentation and labor-intensive manual operations. The authors propose an efficient semantic editing paradigm tailored for 3D Gaussian-based volume visualization, which constructs high-level semantic units through feature-aware clustering and employs a hierarchical refinement pipeline comprising random-walk region propagation, cluster selection, and point-level refinement. Notably, this is the first approach to integrate natural language interaction into VR for volumetric data, enabling multimodal exploration that fuses voice commands, visual feedback, and spatial manipulation. Experiments demonstrate that the method significantly improves selection efficiency and user experience on complex regions in both medical and cosmological datasets while effectively reducing interaction burden.

Abstract

Despite the promise of virtual reality (VR) for intuitive spatial interaction, volume visualization (VolVis) in VR remains constrained by high rendering costs and motion discomfort. Recent advances have shown that representing volumetric scenes with 3D Gaussian splatting enables high-performance rendering, making this representation well-suited for VR. However, existing Gaussian-based scene editing workflows remain limited by slow offline segmentation and fatigue-inducing manual selection. To address these challenges, we present Super-Gaussian, a novel VolVis framework that enhances scene editing and interaction in VR through intuitive 3D Gaussian selection and natural language interaction (NLI). Our approach groups Gaussian primitives into higher-level units via feature-aware clustering, enabling efficient selection of complex volumetric regions, such as tumors in medical images or filaments in cosmological data, without point-by-point interaction. Building on this, we introduce a hierarchical select-and-refine workflow that combines random-walk-based region propagation, cluster selection, and point refinement, allowing users to progressively specify regions of interest with reduced effort. We further support on-the-fly text labeling of selected regions using NLI, allowing users to semantically query, interpret, and manipulate content within a visualization-perception-action loop. By integrating multimodal interaction, including speech, visual feedback, and spatial manipulation in VR, our framework supports intuitive exploration, editing, and scientific analysis of volumetric data. We demonstrate the effectiveness of Super-Gaussian through four case studies, quantitative selection benchmarks against existing Gaussian-based techniques, and system-level evaluations. Implementation details and experiments can be found on the project page: https://smin0136.github.io/super-gaussian-project/

Problem

• volume visualization
• 3D Gaussian splatting
• virtual reality
• scene editing
• interactive selection

Innovation

• 3D Gaussian Splatting
• Natural Language Interaction (NLI)
• Volume Visualization
• Virtual Reality
• Feature-aware Clustering