Differentiable Hierarchical Visual Tokenization

2025-11-04

AI Summary

Vision Transformers rely on fixed-size image patches, neglecting intrinsic spatial structure and semantic hierarchy; moreover, existing approaches struggle to enable dynamic granularity adaptation while preserving compatibility with pretrained models. This paper proposes the first end-to-end differentiable hierarchical visual tokenization method: leveraging an information-theoretic criterion for model selection, it adaptively generates multi-granularity tokens directly from pixels, without requiring fine-tuning to integrate into standard Transformer architectures. The method is natively compatible with mainstream pretrained vision models, enabling plug-and-play upgrades and cross-modal raster-to-vector output generation. Experiments demonstrate state-of-the-art or competitive performance on both image classification and dense prediction tasks. To our knowledge, this is the first differentiable visual tokenization framework that simultaneously achieves semantic awareness, spatial fidelity, and architectural compatibility.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance

Abstract

Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level granularity while remaining backward-compatible with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense-prediction tasks, and even supports out-of-the-box raster-to-vector conversion.

Problem

• Replaces fixed patch tokens with adaptive content-aware tokenization
• Enables pixel-level granularity while maintaining architecture compatibility
• Supports classification, dense prediction, and raster-to-vector conversion

Innovation

• Differentiable tokenizer adapts to image content granularity
• Hierarchical model selection with information criteria
• Backward-compatible with existing pretrained architectures