Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement

2026-08-04

AI Summary

This work addresses the limitation of existing vision-language model self-augmentation methods, which predominantly focus on textual augmentation while neglecting effective image-level enhancement targeting weak visual understanding. To this end, the authors propose the FISA framework, which, for the first time, leverages the model’s own failure cases to generate visually more challenging yet answer-preserving augmented samples. A built-in self-checking mechanism combined with dual-fidelity filtering ensures semantic consistency without requiring external supervision. Trained in conjunction with textual self-augmentation, FISA significantly improves model performance on both in-distribution and out-of-distribution visual question answering benchmarks. Moreover, the synthesized data demonstrates superior efficiency compared to generic image augmentation baselines, validating the method’s effectiveness and compatibility.

Abstract

Multimodal large language models (MLLMs) have achieved remarkable performance across vision-language tasks, but their progress depends heavily on large-scale, high-quality multimodal data that are costly to annotate. Self-augmentation offers a promising alternative by enabling models to expand their own training data without external supervision. However, existing MLLM self-augmentation methods are largely text-centric, while image augmentation remains underexplored and typically relies on generic or handcrafted transformations that are weakly aligned with the model's actual incapability. We propose Failure-informed Image Self-Augmentation (\textbf{FISA}), a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases. Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion. Experiments on visual question answering benchmarks show that the proposed method consistently improves performance across both in-distribution and out-of-distribution settings. Further experiments validate the compatibility of FISA with existing textual self-augmentation approaches, the superior data efficiency of the synthesized samples over generic image augmentation baselines, and the practical effectiveness of the proposed filtering strategy.

Problem

• multimodal large language models
• self-augmentation
• image augmentation
• failure cases
• data efficiency

Innovation

• Failure-informed Augmentation
• Image Self-Augmentation
• Multimodal Large Language Models
• Self-Improvement
• Dual Fidelity Filtering