Mind the Gap! Static and Interactive Evaluations of Large Audio Models

2025-02-21

AI Summary

This study addresses the misalignment between Large Audio Models (LAMs) and real-world user needs. Through 7,500 human voice interaction trials, we identified core usage scenarios via LDA topic modeling and assessed consistency between static benchmarks and user preferences using Spearman’s rank correlation and multi-benchmark regression (R² = 0.30). We propose, for the first time, a user-interaction-centric evaluation paradigm for LAMs. Results reveal weak empirical correlation (τ ≤ 0.33) between prevailing static benchmarks and actual user experience; only two of twenty benchmark datasets—voice-based QA and age prediction—exhibit statistically significant positive correlations with user preference. This demonstrates that no single metric reliably predicts subjective user satisfaction. The work advocates transitioning toward user-driven evaluation frameworks and provides both methodological foundations and empirical evidence to advance principled, human-centered LAM assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: User Experience and Usability

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

As AI chatbots become ubiquitous, voice interaction presents a compelling way to enable rapid, high-bandwidth communication for both semantic and social signals. This has driven research into Large Audio Models (LAMs) to power voice-native experiences. However, aligning LAM development with user goals requires a clear understanding of user needs and preferences to establish reliable progress metrics. This study addresses these challenges by introducing an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions from 484 participants. Through topic modeling of user queries, we identify primary use cases for audio interfaces. We then analyze user preference rankings and qualitative feedback to determine which models best align with user needs. Finally, we evaluate how static benchmarks predict interactive performance - our analysis reveals no individual benchmark strongly correlates with interactive results ($ au leq 0.33$ for all benchmarks). While combining multiple coarse-grained features yields modest predictive power ($R^2$=$0.30$), only two out of twenty datasets on spoken question answering and age prediction show significantly positive correlations. This suggests a clear need to develop LAM evaluations that better correlate with user preferences.

Problem

• Evaluate Large Audio Models interactively
• Align LAM development with user needs
• Assess static benchmarks' predictive power

Innovation

• Interactive LAM evaluation approach
• User preference-based model alignment
• Static and interactive performance correlation