ArtisanGS: Interactive Tools for Gaussian Splat Selection with AI and Human in the Loop

2026-02-10

AI Summary

Extracting and controllably editing objects from 3D Gaussian splatting scenes captured in the wild remains challenging. This work proposes an interactive toolkit that combines AI inference with user guidance to enable flexible and precise local selection and segmentation. The core innovation lies in an interactive 2D-to-3D Gaussian selection propagation mechanism, which leverages an AI-driven mask propagation algorithm in tandem with manual refinement to support arbitrary binary segmentation without requiring additional optimization. Furthermore, the method integrates a custom video diffusion model to achieve high-quality localized editing. Experiments demonstrate that the proposed approach outperforms existing techniques on Gaussian splatting selection tasks and validates its effectiveness and practicality across diverse real-world scenes.

Technology Category

• Computer Vision: Segmentation
• Humans and AI: Interaction Techniques and Devices
• Search and Optimization: Local Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

Representation in the family of 3D Gaussian Splats (3DGS) are growing into a viable alternative to traditional graphics for an expanding number of application, including recent techniques that facilitate physics simulation and animation. However, extracting usable objects from in-the-wild captures remains challenging and controllable editing techniques for this representation are limited. Unlike the bulk of emerging techniques, focused on automatic solutions or high-level editing, we introduce an interactive suite of tools centered around versatile Gaussian Splat selection and segmentation. We propose a fast AI-driven method to propagate user-guided 2D selection masks to 3DGS selections. This technique allows for user intervention in the case of errors and is further coupled with flexible manual selection and segmentation tools. These allow a user to achieve virtually any binary segmentation of an unstructured 3DGS scene. We evaluate our toolset against the state-of-the-art for Gaussian Splat selection and demonstrate their utility for downstream applications by developing a user-guided local editing approach, leveraging a custom Video Diffusion Model. With flexible selection tools, users have direct control over the areas that the AI can modify. Our selection and editing tools can be used for any in-the-wild capture without additional optimization.

Problem

• 3D Gaussian Splat
• interactive selection
• segmentation
• in-the-wild capture
• controllable editing

Innovation

• Gaussian Splatting
• Interactive Segmentation
• AI-Human Collaboration
• 3D Editing
• Video Diffusion Model