🤖 AI Summary
This study addresses the tendency of existing multi-agent policy simulations—based on homogeneous large language models—to generate artificial consensus that fails to capture genuine value disagreements. To overcome this limitation, the authors propose the AI Council, a three-stage deliberation framework that assigns heterogeneous 7–9B parameter models distinct value perspectives and incorporates a state-of-the-art model for consistency validation. Their approach demonstrates, for the first time, that architectural heterogeneity significantly reduces policy choice concentration (e.g., from 70.9% to 46.1% in child welfare and from 46.0% to 22.9% in housing policy, p<0.001). The work further identifies a pervasive trade-off between fidelity and diversity under consistency validation and introduces the “credible tension ratio” as a novel metric to evaluate deliberative capacity in smaller models.
📝 Abstract
Multi-agent deliberation systems using large language models (LLMs) are increasingly proposed for policy simulation, yet they suffer from artificial consensus: evaluator agents converge on the same option regardless of their assigned value perspectives. We present the AI Council, a three-phase deliberation framework, and conduct 120 deliberations across two policy scenarios to test two interventions. First, architectural heterogeneity (assigning a different 7-9B parameter model to each value perspective) significantly reduces first-choice concentration compared to a homogeneous baseline (child welfare: 70.9% to 46.1%, p < 0.001, r = 0.58; housing: 46.0% to 22.9%, p < 0.001, r = 0.50). This contrasts with accuracy-oriented multi-agent debate, where heterogeneity does not reduce convergence, suggesting model diversity operates differently when no objectively correct answer exists. Second, coherence validation (using a frontier model to assess whether each evaluator's reasoning is grounded in its assigned values) reveals a fidelity-diversity tradeoff: on a scenario with a dominant option, it further reduces concentration (46.1% to 40.8%, p = 0.004), but on a scenario with genuinely competitive options, it increases concentration (22.9% to 26.6%, p = 0.96) by amplifying high-coherence evaluators who cluster on one option. This tradeoff may be a general property of multi-agent systems employing quality weighting. We report negative results from three failed Delphi designs, demonstrate that 8B models exhibit binary rather than graded responses to counter-arguments, and propose the trustworthy tension rate as a diagnostic measure of small-model deliberation capabilities.