LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection

2026-07-28

AI Summary

This work addresses the challenge that existing deepfake images, exhibiting minimal visual artifacts, often evade detection by appearance-based methods. To overcome this limitation, the authors propose a multimodal detection framework that integrates RGB semantics with reconstruction residuals. Specifically, they leverage a frozen Stable Diffusion DDIM inversion to generate latent-space consistency residual maps and introduce a Where-What-Why architecture to jointly reason about textual descriptions, artifact locations, and their underlying causes. Image-level discrimination is achieved through fusion of RGB and residual features. The study further innovates by incorporating Group Relative Policy Optimization (GRPO), a reinforcement learning strategy that aligns model reasoning with forensic evidence. The method demonstrates robust cross-generator detection performance on UniversalFakeDetect and achieves state-of-the-art artifact localization on the SynthScars benchmark, with ablation studies confirming the efficacy of residual cues.

Abstract

Recent generative models can produce images with few obvious visual artifacts, weakening detectors and explanations that rely only on surface appearance. We present LaP-Forensics, a multimodal framework that augments RGB semantics with reconstruction-based forensic evidence. A frozen Stable Diffusion DDIM inversion-reconstruction model provides a fixed reconstruction reference, and its residual map measures local compatibility with that reference. Independent projectors encode the RGB image and residual map before a structured Where-What-Why model predicts a textual analysis and an artifact mask.Supervised fine-tuning is followed by Group Relative Policy Optimization (GRPO), whose reward combines mask overlap with output-structure and evidence-reference terms. These text-side terms encourage the model to refer to the consistency map but do not constitute a verifier of free-form textual truth. A separate image-level head fuses RGB and DDIM-residual class features. Experiments show cross-generator detection on UniversalFakeDetect and competitive artifact localization on the official SynthScars benchmark. Controlled cue-construction, inversion-horizon, component, reward-term, and counterfactual analyses support the utility of the residual stream under the evaluated settings, while free-form textual faithfulness and reliability under post-processing remain open limitations.

Problem

• Deepfake Detection
• Multimodal Reasoning
• Latent-Pixel Consistency
• Forensic Evidence
• Generative Models

Innovation

• Latent-Pixel Consistency
• Multimodal Reasoning
• DDIM Inversion
• Residual-based Forensics
• Group Relative Policy Optimization