Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate

2026-06-08

AI Summary

This study investigates the effective evaluation of reasoning quality in multi-agent large language model systems for open-ended tasks without reference answers. By analyzing token-level log probabilities during decoding and integrating a debate-based essay scoring framework (from the ASAP dataset) with LLM-as-judge assessment, the authors find that the confidence of early-generated tokens serves as the strongest predictor of reasoning quality. Furthermore, they identify a systematic asymmetry between supportive and adversarial agent roles in multi-agent interactions. The proposed approach significantly outperforms metrics based on full-sequence statistics, offering a lightweight yet reliable mechanism for estimating reasoning quality in multi-agent systems.

Abstract

Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets. We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique. These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.

Problem

• reasoning quality
• multi-agent LLM
• confidence prediction
• open-ended tasks
• LLM evaluation

Innovation

• early-token confidence
• multi-agent LLM debate
• reasoning quality prediction
• log-probability trajectory
• LLM-as-judge