MMAE: A Massive Multitask Audio Editing Benchmark

2026-06-05

AI Summary

Existing evaluation frameworks for audio editing are fragmented and lack a comprehensive benchmark tailored to general-purpose, instruction-driven editing. This work proposes the first large-scale, multitask evaluation benchmark, encompassing seven audio modalities, six levels of complexity, two granularities, and eight operation types. Through human-AI collaboration, the authors construct 2,000 high-fidelity samples and introduce a novel rubric-based, multidimensional evaluation framework that decomposes free-form tasks into verifiable criteria to precisely assess instruction adherence and contextual consistency. Evaluations of state-of-the-art models reveal exact-match accuracy rates consistently below 5%, approaching zero in complex compositional tasks, thereby exposing critical limitations in current systems’ precision and structural robustness.

Abstract

We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation, interactive editing has rapidly expanded from visual domains, pioneered by models like Nano-banana 2 for images and Gemini-Omni for video, into audio. However, the current evaluation infrastructure lags severely, remaining highly fragmented and restricted to specific subdomains or basic operations. Unlike existing benchmarks that are limited in scope, MMAE extends to a broad spectrum of real-world scenarios, encompassing 7 distinct audio modalities, including sound, speech, music, and their mixtures. Furthermore, we establish a comprehensive taxonomy spanning 6 levels of task complexity, from basic modifications to multi-hop reasoning and multi-round editing, 2 levels of granularity, and 8 distinct operation types. Meticulously curated through human-agent collaboration, MMAE comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework. By decomposing free-form tasks into 17,741 verifiable criteria, this robust rubric-based paradigm enables a precise, multi-dimensional assessment of both instruction following and context consistency. Our extensive evaluation of leading models reveals that current systems remain far from achieving reliable edits. Strikingly, the Exact Match Rate (EMR) consistently falls below 5% and plummets to an absolute 0% in complex, mixed-modality tasks, exposing critical bottlenecks in precise execution and structural robustness. We hope MMAE will serve as a catalyst for future advances in the intelligent creation community, providing a clear diagnostic roadmap and establishing a standardized, long-lasting evaluation paradigm for next-generation audio editing systems.

Problem

• audio editing
• evaluation benchmark
• multitask
• instruction-based
• intelligent creation

Innovation

• audio editing benchmark
• instruction-based editing
• multimodal audio
• rubric-based evaluation
• task complexity taxonomy