Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?

2026-08-07

AI Summary

Existing embedding evaluations primarily focus on discriminative performance or spatial separability, offering limited insight into their effectiveness for downstream generative tasks. This work proposes the Generative Embedding Benchmark (GEB), establishing the first evaluation paradigm centered on generative capability: by freezing embeddings and using them to drive a decoder in answering questions, GEB quantifies the amount of recoverable semantic information encoded within. Evaluated on visual question answering, GEB assesses seven prominent embedding models spanning pure vision and vision-language architectures. Results show that the best vision-language model embedding achieves a score of 65.56 across 900 tests, significantly outperforming text-input and zero-embedding baselines. Natural image content proves more recoverable than scene text, while a reference model using raw images attains 84.30, revealing information bottlenecks invisible to conventional metrics.

Abstract

Embeddings have emerged as a standard representational interface linking foundation models with downstream systems. Most embedding benchmarks assess representations through discriminative tasks or geometric criteria centered on separability in embedding space. However, strong performance on such evaluations does not establish whether content compressed into an embedding remains accessible to a downstream generator. To address this gap, we introduce the Generative Embedding Benchmark (GEB), in which a decoder answers questions using only a frozen embedding and question text, without access to the original image or intermediate visual features. Answer quality under this readout measures generative information: the answer-relevant content recoverable from an embedding. GEB includes a curated visual-question-answering dataset with a 1,800-item development split and a held-out 900-item test split covering natural images, scene text, and visual documents. Using a common decoder and training recipe, we evaluate seven public embedding models in visual-only and vision-language joint modes. On the test set, visual-only scores range from 28.25 to 33.21; with image-question joint encoding, all five VLM-based embedding models score higher, and the best reaches 65.56. Matched embeddings also outperform text-only inputs, zero embeddings, and shuffled embeddings. Natural-image information is much easier to recover than scene text or visual-document information, while a Qwen3-VL-2B reference with access to the original image reaches 84.30. Together, these results show that generative readout exposes information bottlenecks that separability-based evaluation does not capture.

Problem

• generative embedding
• information recovery
• embedding evaluation
• downstream generation
• visual question answering

Innovation

• Generative Embedding Benchmark
• embedding evaluation
• generative readout
• vision-language models
• information recovery