MSVS-VAE: Multi-Scale Anchored VecSet for High-Fidelity 3D Reconstruction

2026-07-27

AI Summary

This work addresses the limitations of existing 3D variational autoencoders (VAEs) in high-fidelity reconstruction, which suffer either from the high computational cost of voxel-based representations or from detail loss due to sparsity and global smoothing in set-based approaches. To overcome these challenges, the authors propose a hierarchical set-based VAE that achieves efficient, high-fidelity reconstruction through progressive densification of anchor-based VecSet latent variables and geometry-aware local decoding. Key innovations include hierarchical point shuffle upsampling for latent densification, an AVS-Conv local aggregation operator replacing global attention, and a multi-scale query decoding mechanism that fuses features across granularities. Experiments demonstrate that the proposed method significantly outperforms state-of-the-art approaches on Objaverse, ABO, and real-world datasets, achieving approximately 10× faster decoding than prior set-based methods and nearly 10× greater model compactness compared to voxel-based baselines.

Abstract

High-fidelity 3D generative modeling increasingly relies on the latent diffusion paradigm, where the reconstruction quality of the underlying 3D VAE becomes a primary bottleneck. Existing approaches largely follow two paradigms: sparse voxel-based representations achieve strong reconstruction quality but incur significant memory and computational overhead, while set-based representations are compact and continuous yet typically lag in fidelity due to latent sparsity and excessive global smoothness. We propose MSVS-VAE, a hierarchical set-based VAE that closes this fidelity gap without sacrificing compactness. Our key idea is to progressively densify anchored VecSet latents via hierarchical point-shuffle upsampling, increasing spatial capacity for fine-grained geometry modeling. To efficiently decode from the densified hierarchy, we replace global cross-attention with AVS-Conv, a geometry-aware local aggregation operator operating within local neighborhoods rather than the exhaustive latent set. We further introduce multi-scale query decoding to fuse coarse-to-fine latent features, where coarse scales provide stable global context, and fine scales refine localized geometry, reducing artifacts from overly local receptive fields. Extensive experiments on Objaverse, ABO, and in-the-wild benchmarks demonstrate that MSVS-VAE consistently outperforms prior set-based and voxel-based VAEs, delivering approximately 10x faster decoding than prior set-based methods and approximately 10x higher compactness than voxel-based baselines.

Problem

• 3D reconstruction
• variational autoencoder
• set-based representation
• voxel-based representation
• reconstruction fidelity

Innovation

• Multi-scale VecSet
• Hierarchical point-shuffle upsampling
• AVS-Conv
• Local geometry-aware aggregation
• Latent diffusion 3D reconstruction