From Cloud to Crowd: Democratizing LLM Service with Decentralized Edge Collaboration for RAG

2026-08-01

AI Summary

This work addresses the limitations of small language models (SLMs) deployed at the edge—namely, restricted knowledge coverage and accuracy—and the high cost, poor scalability, and performance instability of large language models (LLMs) in the cloud. To bridge this gap, the authors propose DEFRAG, a decentralized collaborative retrieval-augmented generation (RAG) framework that, for the first time, integrates knowledge graph compression and hybrid retrieval on edge devices. DEFRAG enables query-adaptive selection of SLMs and dynamic tuning of RAG parameters through joint optimization across heterogeneous edge devices. Experimental results demonstrate that DEFRAG substantially narrows the accuracy gap between SLMs and LLMs while reducing inference costs by 98.4%, increasing peak throughput by 97.8%, and ensuring service stability.

Abstract

The rapid advancement of large language models (LLMs) has increased demand for scalable and cost-effective deployment, especially for mobile and edge devices. Cloud-hosted LLMs are powerful but expensive and difficult to scale due to vendor lock-in and high resource needs, resulting in high expenses and unstable performance under load. Recent efforts focus on deploying small language models (SLMs), distilled or pruned from LLMs, on resource-constrained edge devices to reduce costs and improve scalability. However, edge-based SLMs face limited knowledge coverage and notable accuracy gap compared to cloud-based LLMs. To address this, we present DEFRAG, a decentralized edge collaboration system for retrieval-augmented generation (RAG) that optimizes both retrieval and generation across heterogeneous edge devices. For retrieval, DEFRAG compresses and shares knowledge graphs, using hybrid retrieval to expand knowledge coverage. For generation, DEFRAG introduces an optimizer that adaptively selects SLMs and RAG parameters per query, balancing accuracy and cost. We implement DEFRAG on a heterogeneous edge testbed and evaluate it on benchmark QA datasets. We also test it under mobile route stress, non-uniform data placement, and a domain-specific QA workload. The results show that DEFRAG maintains stable service quality and cost efficiency under these broader settings. Results show that DEFRAG narrows the SLM-LLM accuracy gap, while reducing cost by up to 98.4% and increasing peak throughput by up to 97.8% over centralized services. These findings demonstrate the potential of DEFRAG for democratized LLM services at the edge.

Problem

• large language models
• edge computing
• retrieval-augmented generation
• model accuracy
• cost efficiency

Innovation

• Decentralized Edge Collaboration
• Retrieval-Augmented Generation (RAG)
• Small Language Models (SLMs)
• Adaptive Optimization
• Knowledge Graph Compression