E3DGS: Unified Geometric-Photometric Equivariance for 3D Gaussian Splatting via Color-as-Geometry Embedding

2026-07-16

AI Summary

Existing 3D Gaussian splatting methods struggle to achieve strict SE(3) equivariance that unifies geometry and appearance, as color is typically treated as a signal rather than a geometric entity. This work proposes a “color-as-geometry” perspective by algebraically identifying spherical harmonic coefficients (up to degree ℓ=2) with 3×3 matrices, embedding both geometric and photometric attributes into a unified gl(3) carrier space. Equivariance under joint geometric and photometric transformations is realized through the conjugate action of Wigner-D matrices, enabling a strictly SE(3)-equivariant Gaussian representation without requiring Clebsch–Gordan products. Experiments demonstrate that this approach significantly improves robustness to viewpoint changes and data efficiency in modeling Gaussian worlds conditioned on object appearance and actions.

Abstract

3D Gaussian Splatting (3DGS) captures scenes by coupling explicit geometry (position, covariance) with view-dependent photometry (Spherical Harmonics). However, building $\mathrm{SE}(3)$-equivariant architectures on these primitives presents a fundamental representation bottleneck. Color has been treated as a signal rather than a geometric entity, making it nontrivial to unify symmetry across geometry and appearance as the camera frame changes. While translations are handled by relative coordinates, rotations act heterogeneously across attributes: $μ\mapsto Rμ$, $Σ\mapsto RΣR^\top$, and $f_\ell\mapsto D^\ell(R)f_\ell$. This mismatch complicates strict equivariance, leading existing methods to either discard or flatten SH coefficients, thereby breaking symmetry. We propose a unified solution rooted in representation theory: for SH degrees $\ell\le2$, photometry is algebraically isomorphic to a rank-2 geometric tensor. We prove that the Wigner-$D$ action on these SH coefficients can be exactly reformulated as the conjugation action on $3\times3$ matrices. Leveraging this, we introduce the Unified Matrix Embedding, a lifting that maps all Gaussian attributes into a unified carrier space, $\mathfrak{gl}(3)$. Building on the "Color-as-Geometry" formulation, we present E3DGS, a rigid-body ($\mathrm{SE}(3)$) equivariant architecture that processes 3D Gaussians without Clebsch-Gordan tensor products. Evaluations on object vision and action-conditioned Gaussian world modeling demonstrate that our unified approach yields strong robustness under camera-frame changes and improved data efficiency.

Problem

• 3D Gaussian Splatting
• SE(3) equivariance
• geometric-photometric unification
• spherical harmonics
• representation bottleneck

Innovation

• SE(3) equivariance
• 3D Gaussian Splatting
• Color-as-Geometry
• Spherical Harmonics
• Unified Matrix Embedding