From Talking Words to Sharing Thoughts: Scalable Multi-LLM Aggregation via Structured Message Passing

2026-05-29

AI Summary

This work addresses the high computational overhead, substantial latency, and susceptibility to noise from weak models—termed anchor degradation—in existing large language model (LLM) ensemble methods. It introduces, for the first time, belief propagation on a bipartite factor graph to integrate multiple specialized LLMs at the semantic level through a structured message-passing protocol, eliminating the need for additional inference steps. A novel asymmetric damping mechanism is proposed to shield high-confidence model nodes, effectively mitigating anchor degradation. Evaluated on MMLU, MMLU-Pro, GPQA, and MedMCQA benchmarks, the method reduces token consumption by 97%, decreases API calls by up to 6×, and accelerates inference from minutes to milliseconds, significantly outperforming state-of-the-art multi-agent baselines.

Abstract

The emergence of specialized, domain-tuned Large Language Models (LLMs) has demonstrated that smaller models can achieve expert-level performance in specific tasks, while struggling in out-of-domain settings. Current ensemble methods to combine their complementary expertise primarily rely on iterative re-prompting or cross-model refinement. These approaches suffer from high computational costs and latency because they require repeated LLM inference calls. Furthermore, naive aggregation often leads to anchor corruption, in which noise propagated from weaker models degrades the performance of the most accurate expert. To address these challenges, we propose a framework that integrates model predictions at the semantic layer using a bipartite factor graph. In this architecture, individual LLMs are represented as variable nodes, while a set of check nodes assess their consistency based on diverse epistemic criteria. We develop a message-passing protocol inspired by error-recovery systems to resolve disagreements iteratively. Furthermore, we introduce an asymmetric damping mechanism that protects high-reliability anchor nodes from being overridden by the ensemble majority. Unlike existing methods, our approach operates on output distributions and requires no additional LLM calls during the refinement phase. Evaluating on four benchmarks, including MMLU, MMLU-Pro, GPQA, and MedMCQA, our method demonstrates a 97% reduction in token usage and up to a 6X decrease in API calls, reducing inference time from several minutes to mere milliseconds while consistently outperforming leading multi-agent baselines. These results suggest that graph-based belief propagation is a robust, high-speed, and scalable alternative to the current multi-agent LLM systems. The full pipeline and code will be made public.

Problem

• Multi-LLM Aggregation
• Structured Message Passing
• Anchor Corruption
• Computational Efficiency
• Ensemble Methods

Innovation

• structured message passing
• factor graph
• asymmetric damping
• multi-LLM aggregation
• belief propagation