Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

2026-08-03

AI Summary

This work addresses the limitations of existing reinforcement learning reward mechanisms in audio reasoning tasks, which struggle to jointly account for both process and outcome: outcome-oriented rewards disregard reasoning trajectories, while process-oriented rewards rely on static, coarse-grained human-defined rubrics lacking adaptability. To overcome this, we propose AudioRubrics, a novel framework that, for the first time, automatically generates fine-grained, audio-anchored, and evolvable dynamic rubrics directly from raw audio waveforms and dynamically adjusts reward signals based on the model’s own reasoning trajectory. Integrating reinforcement learning, autoregressive reasoning path modeling, audio embeddings, and dynamic rubric synthesis, AudioRubrics establishes an end-to-end trainable mechanism for rubric generation and evaluation. Experiments demonstrate that AudioRubrics significantly outperforms both open-source and trained baselines across three audio reasoning benchmarks, with stable convergence in reasoning length and enhanced perceptual capabilities confirming the efficacy of audio-anchored supervision.

Abstract

Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.

Problem

• audio reasoning
• reinforcement learning
• adaptive rewards
• rubric-based supervision
• acoustic grounding

Innovation

• AudioRubrics
• reinforcement learning
• audio reasoning
• adaptive rewards
• self-evolving rubrics