Towards Interactive Deepfake Analysis

2025-01-02

AI Summary

Existing deepfake image detection methods suffer from reliance on unimodal discriminative models, absence of standardized evaluation protocols, and scarcity of high-quality training data. This paper introduces a novel interactive paradigm for deepfake analysis grounded in instruction-tuned multimodal large language models (MLLMs). Our key contributions are threefold: (1) DFA-Instruct—a first-of-its-kind, GPT-assisted instruction dataset specifically designed for deepfake analysis; (2) DFA-Bench—a comprehensive benchmark covering detection, classification, and attribution tasks; and (3) DFA-GPT—an open-source, interactive system integrating LoRA for efficient MLLM adaptation. Extensive experiments demonstrate state-of-the-art performance across deepfake detection, forgery-type classification, and artifact description. To our knowledge, this work establishes the first complete open-source framework for interactive deepfake analysis, setting a new baseline for interpretable and human-in-the-loop forensic reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Existing deepfake analysis methods are primarily based on discriminative models, which significantly limit their application scenarios. This paper aims to explore interactive deepfake analysis by performing instruction tuning on multi-modal large language models (MLLMs). This will face challenges such as the lack of datasets and benchmarks, and low training efficiency. To address these issues, we introduce (1) a GPT-assisted data construction process resulting in an instruction-following dataset called DFA-Instruct, (2) a benchmark named DFA-Bench, designed to comprehensively evaluate the capabilities of MLLMs in deepfake detection, deepfake classification, and artifact description, and (3) construct an interactive deepfake analysis system called DFA-GPT, as a strong baseline for the community, with the Low-Rank Adaptation (LoRA) module. The dataset and code will be made available at https://github.com/lxq1000/DFA-Instruct to facilitate further research.

Problem

• Deepfake Detection
• Training Dataset
• Recognition Model Limitations

Innovation

• GPT-based Deep Fake Detection
• DFA-Bench Evaluation Framework
• DFA-GPT with LoRA Adaptation