HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

2026-08-01

AI Summary

This work addresses the multi-dimensional cost optimization challenges in deploying Mixture-of-Experts (MoE) models across geographically distributed heterogeneous edge environments, where communication overhead, computational heterogeneity, GPU-CPU loading latency, queue backlogs, and quantization loss must be jointly managed. The authors propose HetRoute, a novel framework that introduces the first unified cost model to co-optimize expert placement, GPU-CPU residency, and quantization precision during offline deployment, while enabling holistic top-k expert routing during online inference to minimize bottleneck-layer costs. Integrating a cost-coupled deployment algorithm, beam-search-based routing, and queue- and quantization-aware scheduling, HetRoute achieves significant improvements on a 10-node heterogeneous edge testbed: it reduces average inference latency by 59.0%, P99 latency by 58.0%, inter-server traffic by 72.1%, and increases throughput by 2.13×, all while strictly bounding quality degradation within the prescribed budget.

Abstract

Mixture-of-Experts (MoE) models have become a dominant architecture for large-scale AI services, yet deploying them over geo-distributed heterogeneous edge servers remains challenging. When the Top-k activated experts of a token are spread across multiple servers, the optimal routing depends jointly on cross-server link bandwidth, heterogeneous GPU computing capability, GPU-CPU expert loading delay, instantaneous queueing backlog, and replica-level quantization quality loss. Existing distributed inference and MoE serving methods address these factors separately and do not provide a unified framework for online multi-server collaborative routing. In this paper, we propose HetRoute, a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference. HetRoute introduces a unified per-assignment cost model that explicitly captures four cost components: cross-server transmission, GPU-CPU offloading, GPU computation with queueing, and quantization-induced quality penalty. Guided by this model, the offline stage determines expert server placement, GPU-CPU residency, and replica precision through a routing-cost-coupled deployment algorithm, while the online stage routes the Top-k activated expert set as a whole by minimizing the bottleneck layer cost via exact enumeration or beam search. Theoretical analysis establishes fallback feasibility, a bound on the number of participating servers, per-layer optimality for small candidate domains, and online computational complexity. Trace-driven evaluation on three MoE models over a heterogeneous 10-server edge testbed shows that HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%, cuts cross-server traffic by up to 72.1%, and achieves 2.13x throughput improvement compared with representative baselines, while keeping quality degradation within the configured budget.

Problem

• Mixture-of-Experts
• distributed edge inference
• heterogeneous routing
• collaborative routing
• cost-aware

Innovation

• Mixture-of-Experts
• heterogeneous edge computing
• cost-aware routing
• collaborative inference
• quantization-aware deployment