MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models

2026-07-20

AI Summary

This work addresses the limited reasoning capabilities of compact language models (≤4B parameters) under constrained training budgets. The authors propose a multi-agent debate-aware reinforcement learning framework that partitions the model into generator and critic roles, fine-tuning only a small subset of parameters via LoRA. A counterfactual critic advantage mechanism is introduced to align the critic’s optimization objective beyond mere agreement with the generator, enabling more precise credit assignment. Coupled with a lightweight multi-turn reasoning protocol, this approach achieves substantial performance gains with minimal trainable parameters. On five mathematical reasoning benchmarks, it improves the accuracy of DeepSeek-R1-Distill-Qwen-1.5B from 39.9% to 41.9% (+2.0 points, p<0.001), using only 1/16 of the parameters required for full fine-tuning, thereby attaining a Pareto-optimal trade-off between accuracy and parameter efficiency.

Abstract

Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 \, \mathrm{B}$ parameters) trained under limited budgets. We introduce MADA-RL, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean-reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from $39.9 \, \%$ to $41.9 \, \%$ ($+2.0$ points, $p < 0.001$) using $16$ times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.

Problem

• compact models
• parameter-efficient reasoning
• reinforcement learning
• mathematical reasoning
• training cost

Innovation

• counterfactual critic advantage
• parameter-efficient fine-tuning
• multi-agent debate
• LoRA adapters
• role-specialized reinforcement learning