VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text

2024-06-10

AI Summary

This paper introduces the Visual Caption Restoration (VCR) task, addressing the challenge of precisely recovering occluded in-image text from pixel-level visual cues within vision-language models—a problem requiring joint alignment of images, natural language, and embedded textual content, thus transcending the modality fragmentation inherent in conventional OCR or masked language modeling. To this end, the authors formally define the pixel-level cross-modal text restoration problem and construct VCR-Wiki, the first large-scale bilingual benchmark (2.46M samples), featuring controllable difficulty synthesis and fine-grained evaluation. They further propose a multi-granularity masking synthesis pipeline and a vision–language joint alignment modeling framework. Experiments reveal that state-of-the-art multimodal large models underperform humans on VCR by over 40 percentage points, with fine-tuning yielding only marginal gains. The code and dataset are publicly released.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured texts using pixel-level hints within images. This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images. While numerous works have integrated text embedded in images into visual question-answering tasks, approaches to these tasks generally rely on optical character recognition or masked language modeling, thus reducing the task to mainly text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct a dataset for VCR called VCR-Wiki using images with captions from Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and hard split variants. Our results reveal that current vision language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-Wiki and the data construction code to facilitate future research.

Problem

• Restoring obscured text in images using pixel-level hints
• Aligning vision, text, and embedded text modalities effectively
• Addressing limitations of text-based processing in visual tasks

Innovation

• Introduces Visual Caption Restoration (VCR) task
• Generates synthetic images with adjustable caption visibility
• Uses pixel-level hints for occluded text restoration