RELATE: Subjective evaluation dataset for automatic evaluation of relevance between text and audio

2025-06-30

AI Summary

In text-to-audio (TTA) generation, evaluating text–audio relevance has long relied on costly human assessments or objective metrics of questionable validity (e.g., CLAPScore). To address this, we introduce RELATE—the first open-source, human-annotated subjective evaluation dataset for TTA relevance assessment, covering diverse acoustic categories and providing fine-grained relevance scores. Leveraging RELATE, we train an end-to-end deep learning model to predict human relevance judgments automatically. Experiments demonstrate that our model significantly outperforms CLAPScore across all sound categories, achieves consistently high performance, and exhibits strong agreement with human raters (Spearman ρ > 0.72). This work establishes the first standardized subjective benchmark for TTA relevance evaluation and provides a reliable, automated assessment tool—thereby filling two critical gaps in the field.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In text-to-audio (TTA) research, the relevance between input text and output audio is an important evaluation aspect. Traditionally, it has been evaluated from both subjective and objective perspectives. However, subjective evaluation is costly in terms of money and time, and objective evaluation is unclear regarding the correlation to subjective evaluation scores. In this study, we construct RELATE, an open-sourced dataset that subjectively evaluates the relevance. Also, we benchmark a model for automatically predicting the subjective evaluation score from synthesized audio. Our model outperforms a conventional CLAPScore model, and that trend extends to many sound categories.

Problem

• Evaluating text-audio relevance is costly and time-consuming.
• Objective evaluation lacks correlation with subjective scores.
• Proposing RELATE dataset and model for automated relevance scoring.

Innovation

• Constructs RELATE dataset for subjective evaluation
• Benchmarks model predicting subjective scores
• Outperforms CLAPScore in sound categories