🤖 AI Summary
This work addresses the challenge of selecting the optimal inference protocol—direct answering, voting, or debate—and achieving efficient routing under a fixed computational budget for open-source large language models. Under a unified generation-length constraint, the study systematically evaluates greedy decoding, three-sample voting, and two-agent critique-and-revise debate on MuSiQue and GSM8K. It proposes a dynamic routing strategy based on pre-inference signals, revealing that voting entropy predicts debate safety but not its necessity, with many beneficial debates occurring when votes are unanimous yet incorrect—highlighting the limitations of lightweight probing methods in identifying samples requiring debate. Experiments employ Llama-3.1-8B and Mistral-3-8B-Instruct models, combined with entropy thresholds, logistic regression, gradient-boosted trees, and self-critique probes. While ideal routing yields up to a 14-point gain, a simple entropy-threshold controller achieves 1.3–1.7 points, with learned methods offering no significant improvement over this baseline.
📝 Abstract
When should a language model answer directly, sample and vote, or engage in multi-agent debate? Recent work shows voting often explains much of the gain attributed to debate, while selective-debate systems activate deliberation only on uncertain examples. We ask: under a matched ceiling on generated tokens (960 per example), how much per-example routing headroom exists, and how much is recoverable from cheap pre-deliberation signals?
We evaluate greedy decoding, three-sample voting, and a two-agent critique-revise debate on MuSiQue and GSM8K using Llama 3.1 8B Instruct and Ministral 3 8B Instruct. On MuSiQue, an oracle selecting the correct protocol per example gains +14.0 and +13.7 pp over the best fixed one. The best fixed protocol is model- and dataset-dependent: each (model, dataset) cell has a different winner.
This headroom is hard to recover from cheap ex-ante signals. A vote-entropy threshold is the only controller that directionally beats the best fixed protocol on both models (+1.3 and +1.7 pp), though individual paired-bootstrap CIs include zero. A joint analysis (meta-analysis +1.6 pp, p=0.125; Bayesian P(both>0)=0.59) is directionally consistent but not significant. Learned controllers (LR, GBT) do not outperform the threshold.
The key finding is structural: vote entropy predicts where debate is safe, not where debate is needed. High entropy sharply reduces debate backfire, but 66% of debate-helpful examples (31/47) occur when voting is unanimous but wrong. A single-prompt self-critique probe on Llama flips the answer in 127/127 unanimous cases, yielding zero mutual information with the debate-helpful label; we cannot rule out a prompt-compliance artifact, but either interpretation disqualifies the probe as a router. Recovering the remaining headroom requires behavioral probes that avoid format-compliance confounds at the 8B scale.