Technology Category
Natural Language Processing: SpeechMachine Learning: Large Multimodal Models (LMMs)Intelligent Robots: Multimodal Perception & Sensor Fusion
STAR: Speech-to-Audio Generation via Representation Learning
🤖 AI Summary
Existing speech-to-audio generation systems predominantly adopt cascaded architectures, suffering from low inference efficiency and severe error propagation. To address these limitations, we propose STAR—the first end-to-end speech-to-audio generation framework that directly utilizes raw speech as the interactive control signal for audio synthesis. STAR employs deep representation learning to extract sound events and scene semantics from speech, introduces a bridging network to map speech representations to multimodal audio features, and adopts a two-stage training strategy to jointly optimize representation learning and audio synthesis. Experiments demonstrate that STAR reduces speech processing latency by 76.9% compared to cascaded baselines, while significantly outperforming them in audio fidelity, event accuracy, and scene consistency. These results validate the feasibility and superiority of end-to-end, speech-driven audio generation.
Technology Category
Application Category
📝 Abstract
This work presents STAR, the first end-to-end speech-to-audio generation framework, designed to enhance efficiency and address error propagation inherent in cascaded systems. Unlike prior approaches relying on text or vision, STAR leverages speech as it constitutes a natural modality for interaction. As an initial step to validate the feasibility of the system, we demonstrate through representation learning experiments that spoken sound event semantics can be effectively extracted from raw speech, capturing both auditory events and scene cues. Leveraging the semantic representations, STAR incorporates a bridge network for representation mapping and a two-stage training strategy to achieve end-to-end synthesis. With a 76.9% reduction in speech processing latency, STAR demonstrates superior generation performance over the cascaded systems. Overall, STAR establishes speech as a direct interaction signal for audio generation, thereby bridging representation learning and multimodal synthesis. Generated samples are available at https://zeyuxie29.github.io/STAR.
Problem
Research questions and friction points this paper is trying to address.
Developing first end-to-end speech-to-audio generation framework
Addressing error propagation in cascaded speech processing systems
Extracting sound event semantics directly from raw speech signals
Innovation
Methods, ideas, or system contributions that make the work stand out.
End-to-end speech-to-audio generation via representation learning
Bridge network mapping speech representations to audio synthesis
Two-stage training strategy reducing latency by 76.9%
🔎 Similar Papers
No similar papers found.
