🤖 AI Summary
This work proposes a training-free audio style transfer method that achieves high-quality, controllable audio morphing in the discrete latent space of neural audio codecs via residual vector quantization (RVQ). By combining hierarchical grouping of RVQ layers with sequence-level optimization, the approach leverages a user-provided timbral “palette” through a grouped token transfer strategy and a beam search matcher constrained by continuity priors. This enables precise transfer of target timbral details while preserving the rhythmic structure of the source audio. As the first training-free, real-time controllable audio morphing framework operating directly in the RVQ token domain, the method has been successfully deployed as a VST3/AU plugin supporting chunked rendering and an extensible timbre palette, balancing high fidelity with interactive efficiency.
📝 Abstract
Neural audio codecs were originally developed for high-fidelity compression; however, their latent token representations and expressive decoders also constitute a powerful substrate for controllable audio transformation. This work introduces Neural Morphing, a training-free token-domain audio effect that selects residual-vector-quantized (RVQ) token grains from a user palette and decodes the edited stream through a pretrained codec. The method combines an RVQ-group transfer policy that separates coarse, middle, and fine codebook groups with a continuity-constrained sequence matcher that replaces independent greedy selection with bounded beam search. The intended output is a controlled hybrid: the source preserves rhythmic organization while the palette contributes timbral color and residual detail. We focus on the implementation and realtime behavior of a deployable VST3/AU system, including chunked rendering, palette-size scaling, and backend health checks.