PitchFlower: A flow-based neural audio codec with pitch controllability

2025-10-29

AI Summary

To address the low pitch control accuracy and strong acoustic feature coupling in neural audio codecs, this paper proposes a flow-based disentangled speech codec. Methodologically, it introduces explicit F0-conditioned input and applies flattening and random offset perturbations to the F0 contour during training; combined with a vector-quantized bottleneck and a flow-based decoder, it effectively disentangles pitch from other acoustic attributes. Experiments demonstrate that the model achieves synthesis quality comparable to WORLD and SiFiGAN while significantly improving pitch controllability and robustness. Moreover, the architecture provides a scalable framework for further disentangling additional prosodic attributes—such as duration and loudness—enabling fine-grained, attribute-specific speech manipulation.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems

Abstract

We present PitchFlower, a flow-based neural audio codec with explicit pitch controllability. Our approach enforces disentanglement through a simple perturbation: during training, F0 contours are flattened and randomly shifted, while the true F0 is provided as conditioning. A vector-quantization bottleneck prevents pitch recovery, and a flow-based decoder generates high quality audio. Experiments show that PitchFlower achieves more accurate pitch control than WORLD at much higher audio quality, and outperforms SiFiGAN in controllability while maintaining comparable quality. Beyond pitch, this framework provides a simple and extensible path toward disentangling other speech attributes.

Problem

• Developing a neural audio codec with explicit pitch controllability
• Enhancing audio quality while maintaining precise pitch manipulation
• Providing extensible framework for disentangling multiple speech attributes

Innovation

• Flow-based neural audio codec with pitch control
• F0 perturbation and conditioning for disentanglement
• Vector-quantization bottleneck prevents pitch recovery