Fine-Grained Frame Modeling in Multi-head Self-Attention for Speech Deepfake Detection

2026-02-04

AI Summary

This work addresses the challenge that existing deepfake audio detection methods struggle to capture localized and subtle artifacts in temporal sequences. To this end, the authors propose a Transformer-based fine-grained frame modeling approach that incorporates a multi-head voting mechanism to identify critical frames and introduces a cross-layer refinement module to enhance sensitivity to faint forgery cues. Evaluated on the LA21, DF21, and ITW datasets, the method achieves equal error rates of 0.90%, 1.88%, and 6.64%, respectively, substantially outperforming current baselines. These results demonstrate the effectiveness of the proposed mechanisms in accurately modeling localized forged regions within speech signals.

Technology Category

• Natural Language Processing: Speech
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Transformer-based models have shown strong performance in speech deepfake detection, largely due to the effectiveness of the multi-head self-attention (MHSA) mechanism. MHSA provides frame-level attention scores, which are particularly valuable because deepfake artifacts often occur in small, localized regions along the temporal dimension of speech. This makes fine-grained frame modeling essential for accurately detecting subtle spoofing cues. In this work, we propose fine-grained frame modeling (FGFM) for MHSA-based speech deepfake detection, where the most informative frames are first selected through a multi-head voting (MHV) module. These selected frames are then refined via a cross-layer refinement (CLR) module to enhance the model's ability to learn subtle spoofing cues. Experimental results demonstrate that our method outperforms the baseline model and achieves Equal Error Rate (EER) of 0.90%, 1.88%, and 6.64% on the LA21, DF21, and ITW datasets, respectively. These consistent improvements across multiple benchmarks highlight the effectiveness of our fine-grained modeling for robust speech deepfake detection.

Problem

• speech deepfake detection
• fine-grained frame modeling
• multi-head self-attention
• temporal artifacts
• spoofing cues

Innovation

• Fine-Grained Frame Modeling
• Multi-Head Self-Attention
• Speech Deepfake Detection
• Multi-Head Voting
• Cross-Layer Refinement