Reconstructive Visual Instruction Tuning

2024-10-12

arXiv.org

AI Summary

Existing large multimodal models rely predominantly on text-only supervision, limiting their capacity to capture deep visual semantics and resulting in coarse-grained understanding and hallucination-prone behavior. To address this, we propose Reconstruction-based Visual Instruction Tuning (ROSS), a novel latent-space reconstruction paradigm oriented toward visual output: ROSS denoises and reconstructs image embeddings from the SigLIP encoder—bypassing RGB pixel-level regression—to enhance fine-grained visual perception and hallucination robustness. This is synergistically integrated with joint vision–language instruction tuning. ROSS consistently improves performance across diverse vision encoders and large language models. Notably, using only a single SigLIP encoder, ROSS achieves state-of-the-art results on multiple benchmarks, significantly outperforming multi-expert ensemble approaches while maintaining architectural simplicity and training efficiency.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.

Problem

• Multi-modal Models
• Image Processing
• Deep Content Understanding

Innovation

• ROSS
• Multi-modal Models
• Image Reconstruction