ADIFF: Explaining audio difference using natural language

2025-02-06

AI Summary

This paper formally introduces and defines the novel task of *Natural Language Explanation of Audio Differences*, which aims to generate precise textual descriptions of disparities between two audio clips across dimensions including acoustic events, scenes, spectral characteristics, and affective impact—supporting applications in audio forensics, quality assessment, and generative audio evaluation. To this end, we construct the first dual-level benchmark dataset enabling three-tiered explanations: event-level → semantic-level → emotion-semantic fused level. Our method employs an audio encoder to extract dual-clip embeddings, integrated with a large language model via a lightweight adapter comprising a cross-modal cross-projection module, a position-aware captioning mechanism, and a three-stage prefix fine-tuning paradigm for fine-grained difference modeling. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art models (e.g., Qwen-Audio) on both automatic metrics and human evaluation, achieving breakthrough improvements—particularly in Tier-3 emotion-semantic explanation.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scenes, signal characteristics, and their emotional impact on listeners. This paper stands out as the first work to comprehensively study the task of explaining audio differences and then propose benchmark, baselines for the task. First, we present two new datasets for audio difference explanation derived from the AudioCaps and Clotho audio captioning datasets. Using Large Language Models (LLMs), we generate three levels of difference explanations: (1) concise descriptions of audio events and objects, (2) brief sentences about audio events, acoustic scenes, and signal properties, and (3) comprehensive explanations that include semantics and listener emotions. For the baseline, we use prefix tuning where audio embeddings from two audio files are used to prompt a frozen language model. Our empirical analysis and ablation studies reveal that the naive baseline struggles to distinguish perceptually similar sounds and generate detailed tier 3 explanations. To address these limitations, we propose ADIFF, which introduces a cross-projection module, position captioning, and a three-step training process to enhance the model's ability to produce detailed explanations. We evaluate our model using objective metrics and human evaluation and show our model enhancements lead to significant improvements in performance over naive baseline and SoTA Audio-Language Model (ALM) Qwen Audio. Lastly, we conduct multiple ablation studies to study the effects of cross-projection, language model parameters, position captioning, third stage fine-tuning, and present our findings. Our benchmarks, findings, and strong baseline pave the way for nuanced and human-like explanations of audio differences.

Problem

• Explains audio differences using natural language.
• Proposes benchmarks for audio difference explanation.
• Enhances model to generate detailed audio descriptions.

Innovation

• Generates multi-level audio explanations
• Introduces cross-projection module
• Uses three-step training process