Safeguarding Facial Identity against Diffusion-based Face Swapping via Cascading Pathway Disruption

2026-01-21

AI Summary

This work addresses the critical threat posed by diffusion-based face swapping to facial identity privacy, a challenge inadequately mitigated by existing defenses. The authors propose a systematic cascaded perturbation strategy that, for the first time, leverages structural robustness and static conditional guidance mechanisms to inject targeted disturbances at key bottlenecks within the identity pathway, thereby globally disrupting the generation process. The method integrates identity localization interference, identity erasure, attention disentanglement, and contamination of intermediate diffusion features, coupled with a perceptually adaptive adversarial search in the latent manifold. Extensive experiments demonstrate that the proposed approach significantly outperforms current defense techniques across multiple diffusion-based face swapping models, yielding adversarial examples that achieve high attack efficacy while remaining visually imperceptible.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

The rapid evolution of diffusion models has democratized face swapping but also raises concerns about privacy and identity security. Existing proactive defenses, often adapted from image editing attacks, prove ineffective in this context. We attribute this failure to an oversight of the structural resilience and the unique static conditional guidance mechanism inherent in face swapping systems. To address this, we propose VoidFace, a systemic defense method that views face swapping as a coupled identity pathway. By injecting perturbations at critical bottlenecks, VoidFace induces cascading disruption throughout the pipeline. Specifically, we first introduce localization disruption and identity erasure to degrade physical regression and semantic embeddings, thereby impairing the accurate modeling of the source face. We then intervene in the generative domain by decoupling attention mechanisms to sever identity injection, and corrupting intermediate diffusion features to prevent the reconstruction of source identity. To ensure visual imperceptibility, we perform adversarial search in the latent manifold, guided by a perceptual adaptive strategy to balance attack potency with image quality. Extensive experiments show that VoidFace outperforms existing defenses across various diffusion-based swapping models, while producing adversarial faces with superior visual quality.

Problem

• face swapping
• diffusion models
• identity protection
• privacy
• adversarial defense

Innovation

• cascading pathway disruption
• diffusion-based face swapping
• identity erasure
• attention decoupling
• latent adversarial perturbation