Beyond Global Latents: Chunk-Based Sparse Grid VAE for Scalable 3D Modeling

2026-08-03

AI Summary

This work addresses the scalability limitations of high-resolution sparse voxel grids, whose memory consumption grows rapidly with the number of active cells. To overcome this challenge, the authors propose ChunkVAE, a block-based variational autoencoder for sparse grids that confines encoding and decoding operations to local blocks of variable size, enabling efficient inference. The method introduces localized learning operators combined with balanced binary object partitioning and S-curve weighted blending to effectively suppress inter-block boundary artifacts. It further decouples block sizes between training and inference and supports parallel processing. Experiments demonstrate that ChunkVAE matches or exceeds state-of-the-art performance across three benchmarks, scales to resolutions from 512³ up to 1536³, significantly reduces peak memory usage, accelerates inference, and improves image-to-3D generation quality.

Abstract

Sparse voxel grids preserve the spatial structure needed for detailed 3D reconstruction, but their memory still grows rapidly with resolution as active surface cells increase. We introduce ChunkVAE, a sparse grid variational autoencoder organized around local chunks rather than a global latent volume. Local learned operators permit independently chosen encoder and decoder partitions and allow inference chunk sizes to differ from training. Two complementary data operators make this flexibility practical: Balanced Binary Object Partitioning distributes active cells while limiting replicated overlap, while S-Curve weighted stitching attenuates unreliable boundary features when assembling a global latent or reconstruction. Across three object benchmarks, ChunkVAE is competitive with or better than strong baselines from $512^3$ to $1536^3$; smaller chunks lower peak allocated memory and shorten per-chunk compute, enabling faster parallel inference. Stable stitched latents and improved image to 3D metrics indicate that local compression can scale geometry while retaining the global interface required downstream.

Problem

• sparse voxel grids
• scalable 3D modeling
• memory efficiency
• high-resolution reconstruction
• 3D geometry compression

Innovation

• ChunkVAE
• sparse voxel grid
• local chunk
• variational autoencoder
• scalable 3D modeling