Problem
Research questions and friction points this paper is trying to address.
joint embedding
audio effect code
digital signal processing
audio-to-code retrieval
music information retrieval
PEACE: Joint Embeddings of DSP Effects Code and Audio
🤖 AI Summary
This study addresses the absence of joint embeddings between audio effect code and output audio, which hinders the simultaneous capture of DSP topology and parameters. To this end, we propose the first Faust code-audio joint embedding framework. Methodologically, it employs an AFx-Rep audio encoder, a fine-tuned T5 Transformer, and a message-passing graph neural network to process intermediate representations. By masking UI parameters, the framework decouples effect chain topology and achieves cross-modal alignment retrieval through multimodal objectives. Experimental results demonstrate that our approach significantly outperforms existing pretrained models on an out-of-distribution reverb retrieval benchmark. It effectively enhances frozen audio representations and successfully recovers ordered chain topologies, thereby overcoming the limitations of conventional supervised methods.
📝 Abstract
This paper introduces PEACE, the first joint embedding of audio effect code and output audio. Building on SLAP's multimodal objective, we pair an AFx-Rep audio encoder with two code encoders for Faust, a functional language for audio signal processing. First, we evaluate a fine-tuned T5 transformer over Faust source code. Second, we evaluate a message-passing graph neural network over an intermediate representation of the Faust compiler, capturing both topology and UI parameters. We evaluate on audio-to-code retrieval, where masking UI parameters at inference yields embeddings that encode effect chain topology alone. When parameters are visible, the two code encoders tie on retrieval of mixed-length chains but have tradeoffs on single-effect galleries. With parameters fully masked, PEACE recovers ordered chain topology far above chance without the limitations of supervised methods. PEACE outperforms pretrained models on an out-of-distribution reverb retrieval benchmark and can improve frozen audio-only representations. Its dual understanding of topology and parameters lays the groundwork for music information retrieval systems that search, generate, and condition on DSP code.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Joint Embedding
Audio Effects Code
Graph Neural Network
Audio-to-Code Retrieval
Digital Signal Processing
🔎 Similar Papers
No similar papers found.
