Problem
Research questions and friction points this paper is trying to address.
visual-text compression
evaluation framework
text preservation
multimodal large language models
semantic decoupling
Decoupling semantics from vision: A framework for faithful visual-text compression evaluation
🤖 AI Summary
This work addresses the limitations of existing vision-text compression (VTC) evaluation protocols, which rely on downstream task performance and are thus confounded by the strong semantic priors inherent in multimodal large language models, making it difficult to accurately assess textual fidelity. To overcome this, we propose the first framework that decouples semantic reasoning from compression quality assessment. By introducing the ZeroSense Benchmark—a dataset specifically designed with minimal semantic correlation between visual and textual modalities—we eliminate dependence on textual semantics and enable unbiased evaluation of VTC fidelity. Experimental results demonstrate a significant divergence between VTC fidelity and downstream task performance, underscoring the necessity and effectiveness of our approach in faithfully reflecting true compression quality.
📝 Abstract
Recent visual-text compression (VTC) methods, typified by DeepSeek-OCR, report impressive high token compression ratios for long-context modeling tasks by leveraging text-to-image rendering. However, existing evaluation protocols heavily rely on downstream task performance. Such evaluation metrics fail to accurately measure text preservation due to the strong inherent linguistic priors of Multimodal Large Language Models (MLLMs). In this work, we introduce a new evaluation framework that decouples MLLMs' capabilities to faithfully assess VTC quality. Within this framework, we further introduce the ZeroSense Benchmark to ensure low semantic correlation of testing samples. By eliminating textual dependencies, our benchmark guarantees that the evaluation results are purely reflective of VTC quality, unaffected by the semantic inference capabilities of downstream models. Extensive experiments across multiple datasets demonstrate that VTC quality and downstream task accuracy diverge significantly, highlighting the necessity of our decoupled evaluation framework.
Innovation
Methods, ideas, or system contributions that make the work stand out.
visual-text compression
decoupled evaluation
ZeroSense Benchmark
semantic priors
faithful compression assessment
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
