AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models

2026-09-30

AI Summary

This study addresses the vulnerability of large audio-language models to input perturbations and their tendency to hallucinate. To mitigate these issues, this work proposes AnchorPrompt, a method that learns decoder soft prompt vectors via self-distillation while keeping model parameters frozen, thereby enhancing robustness. Notably, AnchorPrompt achieves zero-shot generalization to unseen distortions at inference without requiring explicit perturbation detection. By targeting clean-recording predictions, it suppresses hallucinations and optimizes refusal mechanisms. The training procedure integrates parameter-efficient fine-tuning with diverse data augmentation strategies. Empirical evaluations across multiple benchmarks demonstrate that the proposed approach significantly improves answer consistency and effectively reduces hallucinations under severe corruptions, all while incurring negligible degradation in accuracy on clean data.

Abstract

Large audio-language models (LALMs) are sensitive to input perturbations, such as noise, waveform corruption, and adversarial injections. We propose AnchorPrompt, an efficient adaptation method that keeps the model frozen and learns a single block of prompt vectors inserted at the decoder input, between the audio and question embeddings. We train these vectors through self-distillation over diverse audio and text perturbations. To improve answer consistency and mitigate hallucination, we use the model's prediction on the clean recording as the target for answerable inputs, and assign a refusal target when the audio lacks sufficient evidence to answer. Furthermore, AnchorPrompt is perturbation-agnostic at inference, requiring no prior detection of perturbations and enabling zero-shot transfer to unseen distortions. We evaluate three LALMs across three benchmarks and show that AnchorPrompt improves answer consistency in most tested conditions. Clean accuracy improves in six of nine model-benchmark pairs, with minimal impact on the remainder of 1.2% at most. Crucially, AnchorPrompt reduces hallucinations under severe audio corruption while keeping false refusals on clean audio rare. Finally, these consistency gains transfer to unseen perturbations, such as choice permutations and reverberation.

Problem

• audio-language models
• input perturbations
• hallucination
• robustness
• answer consistency

Innovation

• Self-Distillation
• Soft Prompts
• Audio-Language Models
• Hallucination Mitigation
• Perturbation-Agnostic