Distributed Mixture-of-Agents for Edge Inference with Large Language Models

2024-12-30

AI Summary

To address the challenges of resource-constrained edge devices and decentralized collaborative inference among multiple lightweight LLMs, this paper proposes the first fully decentralized edge Mixture-of-Agents (MoA) inference framework. It employs a gossip-based protocol for asynchronous peer-to-peer communication among devices and introduces a distributed queue model, accompanied by rigorous stability analysis—yielding the first provable sufficient condition for queue boundedness. The framework supports user-specific model loading and prompt-enhanced collaborative response generation. Evaluated on AlpacaEval 2.0, the optimal configuration achieves a +12.3% win rate over single-model baselines while guaranteeing strict boundedness of request queue length. This work establishes a novel paradigm for low-latency, high-robustness collaborative LLM inference at the edge.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Multiagent Systems: Agent Communication
• Search and Optimization: Distributed Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Mixture-of-Agents (MoA) has recently been proposed as a method to enhance performance of large language models (LLMs), enabling multiple individual LLMs to work together for collaborative inference. This collaborative approach results in improved responses to user prompts compared to relying on a single LLM. In this paper, we consider such an MoA architecture in a distributed setting, where LLMs operate on individual edge devices, each uniquely associated with a user and equipped with its own distributed computing power. These devices exchange information using decentralized gossip algorithms, allowing different device nodes to talk without the supervision of a centralized server. In the considered setup, different users have their own LLM models to address user prompts. Additionally, the devices gossip either their own user-specific prompts or augmented prompts to generate more refined answers to certain queries. User prompts are temporarily stored in the device queues when their corresponding LLMs are busy. Given the memory limitations of edge devices, it is crucial to ensure that the average queue sizes in the system remain bounded. In this paper, we address this by theoretically calculating the queuing stability conditions for the device queues under reasonable assumptions, which we validate experimentally as well. Further, we demonstrate through experiments, leveraging open-source LLMs for the implementation of distributed MoA, that certain MoA configurations produce higher-quality responses compared to others, as evaluated on AlpacaEval 2.0 benchmark. The implementation is available at: https://github.com/purbeshmitra/distributed_moa.

Problem

• Distributed Computing
• Edge Devices
• Collaborative Language Modeling

Innovation

• Mixture of Agents (MoA)
• Decentralized Collaboration
• Edge Computing Language Models