Beyond Prompt Adherence: Auditing Attribute-Level Voice Control in Speech Generation

2026-08-01

AI Summary

This work addresses a critical gap in the evaluation of controllable speech generation systems, which typically focus on target prompt alignment while neglecting the stability of non-target attributes. The study systematically reveals, for the first time, that adjusting a target attribute often induces unintended shifts in other acoustic or speaker characteristics. To investigate this phenomenon, the authors construct an audit dataset comprising 5,940 samples and perform controlled pairwise evaluations using multidimensional metrics spanning acoustics, prosody, content, and speaker traits. They further propose VoDER-Cal, a training-free, inference-time candidate re-ranking method that significantly enhances attribute preservation. In a three-candidate setting, VoDER-Cal improves joint success rate from 4.8% to 14% and reduces non-target deviation from 0.344 to 0.276, demonstrating its effectiveness in mitigating unintended perturbations during targeted voice editing.

Abstract

Natural-language descriptions have become a flexible interface for controlling generated speech. Existing evaluations largely assess whether an output matches a prompt, but prompt matching alone does not reveal whether characteristics outside the intended change remain stable. We examine this distinction through a controlled paired audit of three speech-generation systems: CosyVoice3, VoxCPM2, and Fish-Speech-S2. The evaluation contains 5,940 outputs spanning six reference speakers, ten texts, three random seeds, and eleven conditions. Using acoustic, prosodic, content, and speaker measurements, we find that responses in the expected target direction are frequently accompanied by changes outside descriptor-specific signal-level target sets. This pattern remains among outputs whose target response exceeds baseline seed variation, and the accompanying changes differ substantially across systems. We further introduce VoDER-Cal, a training-free candidate selector that retains sufficiently strong target responses while favoring smaller off-target deviations. A three-candidate pool raises the joint success rate from 4.8% under single-sample direct generation to approximately 14% for all candidate-selection policies. Within the matched three-candidate budget, VoDER-Cal reduces held-out off-target deviation from 0.344 under target-only selection to 0.276 and improves listener-rated preservation. Preservation-sensitive evaluation therefore complements prompt-adherence evaluation, while candidate reranking offers a practical inference-time improvement. Code, configuration files, and analysis scripts are available at https://github.com/intelland/VoDER

Problem

• speech generation
• attribute control
• off-target deviation
• prompt adherence
• voice preservation

Innovation

• attribute-level control
• off-target deviation
• speech generation auditing
• candidate reranking
• VoDER-Cal