Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models

2024-12-16

arXiv.org

AI Summary

Existing image protection methods struggle to simultaneously achieve strong adversarial robustness, visual imperceptibility, and low inference latency—hindering practical deployment. To address the misuse of diffusion models (e.g., artistic plagiarism and deepfakes), this paper proposes a lightweight, highly stealthy real-time image protection framework. Our method introduces perturbation pretraining and a dynamic hybrid perturbation mechanism, jointly optimizing the protection loss across multiple latent feature spaces of a Variational Autoencoder (VAE). Additionally, we design an adaptive target-attack modeling strategy during inference to enhance transferability and resilience against unknown adversaries. Extensive experiments demonstrate that our approach achieves state-of-the-art protection performance: it reduces perceptual distortion by 42%, cuts inference latency by 90%, and significantly improves robustness and practicality—enabling end-to-end real-time deployment on resource-constrained devices.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advancements in diffusion models revolutionize image generation but pose risks of misuse, such as replicating artworks or generating deepfakes. Existing image protection methods, though effective, struggle to balance protection efficacy, invisibility, and latency, thus limiting practical use. We introduce perturbation pre-training to reduce latency and propose a mixture-of-perturbations approach that dynamically adapts to input images to minimize performance degradation. Our novel training strategy computes protection loss across multiple VAE feature spaces, while adaptive targeted protection at inference enhances robustness and invisibility. Experiments show comparable protection performance with improved invisibility and drastically reduced inference time. The code and demo are available at url{https://webtoon.github.io/impasto}

Problem

• Protect images from misuse by diffusion models
• Balance protection efficacy, invisibility, and latency
• Enhance robustness with adaptive targeted protection

Innovation

• Perturbation pre-training reduces latency
• Mixture-of-perturbations adapts to input images
• Multi-VAE protection loss enhances robustness