dRAE: Representation Autoencoder with Hyper-Spherical Codes

2026-07-24

AI Summary

This work addresses the challenge of effectively discretizing high-dimensional visual representations for integration with language models. Existing quantization methods suffer from metric mismatch between Euclidean distance and the anisotropic structure of representation spaces, leading to codebook collapse, semantic distortion, and poor scalability. To resolve this, the authors propose Hyperspherical Quantization (HSQ), which decouples semantic content from feature magnitude via angular routing and forms a discrete Representation AutoEncoder (dRAE). HSQ is the first method to reconcile this metric mismatch, enabling semantics-driven code assignment and achieving 100% codebook utilization. It maintains high performance even when scaling the codebook to 131,072 entries, significantly simplifies training, and demonstrates strong results in both visual understanding and generation tasks.

Abstract

In this work, we aim to discretize the high-dimensional visual representations to bridge the gap with language models - a non-trivial challenge, as existing quantization methods suffer from codebook collapse, failing to scale while preserving semantic coherence. We identify the root cause as metric mismatch: standard Euclidean codebook objectives are fundamentally misaligned with the anisotropic geometry of representation space, leading to codebook embeddings with high-variance magnitude scales and uneven angular distributions that hinder scalability. To address this, we propose Hyper-Spherical Quantization (HSQ), which decouples semantic content from feature magnitude via angular routing, preventing code assignment from being dominated by scale rather than meaning. The resulting discrete Representation Autoencoder (dRAE) achieves high-fidelity reconstruction while preserving semantic integrity and supporting scalable codebook budget. Extensive experiments demonstrate consistent performance gains as the vocabulary size scales to 131{,}072, along with 100\% codebook utilization, simplified training pipeline, and strong performance across understanding and generation tasks.

Problem

• discretization
• codebook collapse
• semantic coherence
• representation space
• quantization

Innovation

• Hyper-Spherical Quantization
• Representation Autoencoder
• Codebook Collapse
• Angular Routing
• Discrete Representation