Transition-Aware Backend Dispatch for Edge LLM Inference

2026-07-19

AI Summary

This work addresses the challenges of deploying large language models on edge devices, where inference performance is constrained by model size and significant performance variations across backends due to tensor shape differences. Static scheduling fails to adapt to runtime dynamics, while per-operator backend selection incurs prohibitive switching overhead. To overcome these limitations, the authors propose a dynamic backend scheduling strategy that jointly models operator shape characteristics and cross-backend switching context for the first time, thereby preserving shape-aware performance gains while minimizing switching costs. Evaluation across multiple backends—PyTorch (CPU/CUDA) and ONNX Runtime (CPU)—demonstrates that, over 9,584 operator instances, the approach reduces average latency by 17.4%, energy consumption by 14.4%, and energy-delay product by 28.5% compared to the best static strategy, with consistent generalization improvements observed across six to seven Transformer-based models.

Abstract

Efficient large language model (LLM) inference on edge platforms is limited not only by model size, but also by shape-dependent performance differences across execution backends. Static backend assignment cannot exploit this variation, while independent per-operator selection can introduce costly device and framework switches. This paper presents a transition-aware backend dispatch approach for edge transformer inference. The approach combines current operator features with the previously selected backend to preserve beneficial shape-specific choices while avoiding unnecessary transitions. Ordered traces are collected from full-model inference runs of seven transformer models, and four common operator classes are benchmarked across PyTorch eager CPU, PyTorch eager CUDA, and ONNX Runtime CPU on an NVIDIA Jetson platform. The dispatch policies are evaluated through measurement-backed trace replay using observed operator costs and transition costs measured from actual backend switches. Supported operators are selected dynamically, while operators outside the dispatch scope retain a static assignment. Across 9,584 ordered operator instances and 278 exact shape groups, transition-aware dispatch reduces replayed latency, energy, and energy-delay product relative to the best static policy by 17.4%, 14.4%, and 28.5% on average, respectively. It also reduces switching relative to operator-local selection. Leave-one-model-out evaluation improves all three objectives for six of seven held-out models and improves energy for all seven. These results demonstrate that incorporating operator shape and backend-transition context can improve selective backend dispatch for edge transformer workloads.

Problem

• edge LLM inference
• backend dispatch
• operator shape
• transition cost
• performance variation

Innovation

• transition-aware dispatch
• edge LLM inference
• backend selection
• operator shape
• energy-delay product