FARI: Robust One-Step Inversion for Watermarking in Diffusion Models

2026-07-29

AI Summary

Existing diffusion model watermark inversion methods struggle to balance efficiency and robustness. This work proposes a fast asymmetric robust inversion framework that, for the first time, leverages the low-curvature property of inversion trajectories to enable highly efficient approximation with low NFE (number of function evaluations). By integrating single-step inversion with lightweight adversarial LoRA fine-tuning, the method directly optimizes robustness within an end-to-end adversarial training pipeline. Requiring only approximately 20 minutes of fine-tuning on a single GPU, the approach achieves watermark verification robustness surpassing that of 50-step DDIM inversion while significantly reducing inference latency, thereby overcoming the traditional limitation of high computational cost.

Abstract

Inversion-based watermarking is a promising approach to authenticate diffusion-generated images, yet practical use is bottlenecked by inversion that is both slow and error-prone. While the primary challenge in the watermarking setting is robustness against external distortions, existing approaches over-optimize internal truncation error, and because that error scales with the sampler step size, they are inherently confined to high-NFE (number of function evaluations) regimes that cannot meet the dual demands of speed and robustness. In this work, we have two key observations: (i) the inversion trajectory has markedly lower curvature than the forward generation path does, making it highly compressible and amenable to low-NFE approximation; and (ii) in inversion for watermark verification, the trade-off between speed and truncation error is less critical, since external distortions dominate the error. A faster inverter provides a dual benefit: it is not only more efficient, but it also enables end-to-end adversarial training to directly target robustness, a task that is computationally prohibitive for the original, lengthy inversion trajectories. Building on this, we propose \textbf{FARI} (\textbf{F}ast \textbf{A}symmetric \textbf{R}obust \textbf{I}nversion), a one-step inversion framework paired with lightweight adversarial LoRA fine-tuning of the denoiser for watermark extraction. While consolidation slightly increases internal error, FARI delivers large gains in both speed and robustness: with approximately 20 minutes of fine-tuning on a single NVIDIA RTX A6000 GPU, it surpasses 50-step DDIM inversion on watermark-verification robustness while dramatically reducing inference time. Code and pretrained models are available at https://github.com/0xD009/FARI.

Problem

• inversion-based watermarking
• diffusion models
• robustness
• speed
• external distortions

Innovation

• Fast Inversion
• Robust Watermarking
• Diffusion Models
• Adversarial LoRA
• One-Step Inversion