NoiseCutMix: A Novel Data Augmentation Approach by Mixing Estimated Noise in Diffusion Models

2025-08-30

AI Summary

Existing image-level mixing methods (e.g., CutMix) suffer from unnatural boundaries and poor semantic consistency in high-resolution fused images due to contextual discrepancies between mixed regions. To address this, we propose NoiseCutMix—the first adaptation of CutMix to diffusion models—performing cross-class mixing in the noise estimation space rather than the pixel space. Specifically, NoiseCutMix leverages pre-trained diffusion models (e.g., Stable Diffusion) to extract denoising residuals from two input samples, applies a CutMix mask to locally weight and blend these residuals, and then reconstructs high-fidelity images via reverse diffusion sampling. Crucially, by operating in the latent noise space, NoiseCutMix avoids pixel-level stitching artifacts and enables semantically coherent feature fusion across classes. Extensive experiments demonstrate that NoiseCutMix significantly improves generalization performance of classification models, consistently outperforming conventional data augmentation and random generation baselines across multiple benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

In this study, we propose a novel data augmentation method that introduces the concept of CutMix into the generation process of diffusion models, thereby exploiting both the ability of diffusion models to generate natural and high-resolution images and the characteristic of CutMix, which combines features from two classes to create diverse augmented data. Representative data augmentation methods for combining images from multiple classes include CutMix and MixUp. However, techniques like CutMix often result in unnatural boundaries between the two images due to contextual differences. Therefore, in this study, we propose a method, called NoiseCutMix, to achieve natural, high-resolution image generation featuring the fused characteristics of two classes by partially combining the estimated noise corresponding to two different classes in a diffusion model. In the classification experiments, we verified the effectiveness of the proposed method by comparing it with conventional data augmentation techniques that combine multiple classes, random image generation using Stable Diffusion, and combinations of these methods. Our codes are available at: https://github.com/shumpei-takezaki/NoiseCutMix

Problem

• Combining estimated noise from two classes in diffusion models
• Generating natural high-resolution images with fused characteristics
• Overcoming unnatural boundaries in traditional CutMix augmentation

Innovation

• Mixes estimated noise from two classes
• Uses diffusion models for natural boundaries
• Enhances CutMix with high-resolution generation