InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation

2026-07-21

AI Summary

This work addresses the high computational cost and semantic inconsistency inherent in existing mixup-based data augmentation methods, which often impair model generalization by blending across samples. To overcome these limitations, the authors propose a single-sample, semantically coherent strong augmentation strategy: key regions are first identified via lightweight multi-scale saliency detection, then edited using instruction-guided generative models, and finally recombined with the non-salient parts of the original image, while adaptive fractal structures are injected to enhance representational diversity. Theoretical analysis grounded in second-order neighborhood risk reveals the model’s invariance to generated perturbations and its curvature-suppression mechanism. Evaluated across seven benchmarks, the method consistently outperforms nine state-of-the-art augmentation techniques, achieving superior performance in classification, robustness, calibration, and transfer learning tasks across CNNs, Vision Transformers, and vision-language models.

Abstract

In image and video technologies, data augmentation is widely used to improve the generalization of deep visual models, and mixup-based strategies that interpolate between samples have become the dominant approach. However, computing informative mixing regions adds substantial overhead, and blending content across different images frequently disrupts the semantic integrity of the resulting sample. We propose \our{}, a data augmentation method that constructs challenging yet label-consistent training samples entirely within a single visual sample. \our{} first extracts multi-scale salient patches from the sample using a lightweight saliency detector, refines each patch with an instruction-guided generative model, and blends the edited patch back into the non-salient regions of the same sample; because the generative edits are computed once and cached offline, this step adds negligible training cost. To further diversify the learned representation, \our{} injects self-similar fractal structure into the same salient regions at an adaptive ratio, so each training sample carries both fractal and non-fractal structure. We derive a second-order approximation of the resulting vicinal risk, showing that the method simultaneously enforces invariance to the generative edit and suppresses curvature along the perturbed salient directions, and we verify both predictions empirically. We evaluate on small to large backbones for instance Convolutional Neural Networks (CNNs), Vision Transformers (ViTs) and Vision-Language Foundational Models (VLMs) across seven benchmarks covering coarse- and fine-grained classification, robustness to corruption and occlusion, calibration, and transfer and self-supervised learning, InstructMixup outperforms nine competing augmentation methods, surpassing the strongest baseline across all benchmarks.

Problem

• data augmentation
• mixup
• semantic integrity
• computational overhead
• visual models

Innovation

• instruction-guided generation
• salient patch editing
• within-sample augmentation
• fractal structure injection
• vicinal risk approximation