HorizonServe: Coordinating Request Scheduling with GPU Sharing for Omni-Model Serving

2026-08-03

AI Summary

This work addresses the challenge in unified multimodal model serving where heterogeneous output modalities create conflicts between shared backbone computation and divergent generation stages, hindering existing systems from effectively coordinating temporal and spatial resource multiplexing. To resolve this, the authors propose HorizonServe, the first system to enable joint scheduling of shared and forked phases on a single GPU. HorizonServe integrates SLO-aware request admission control, latency-profile-guided dynamic throttling of streaming multiprocessor (SM) resources, and an execution path rotation mechanism to simultaneously meet diverse first-response latency targets and optimize resource efficiency. Experiments across three omni-model workloads and two GPU platforms demonstrate that HorizonServe improves SLO compliance by up to 7.0× and reduces first-response latency by 38.4%–63.7%.

Abstract

Omni models unify text, speech, image, and multimodal reasoning in a single serving backend, but this unified deployment exposes a new scheduling problem. Requests with different output modalities may share an initial multimodal backbone and then diverge into downstream generation stages, creating heterogeneous first-response metrics and service-level objective (SLO) targets on the same GPU. Existing large language model (LLM) and multimodal serving systems mainly optimize token progress or input-side processing, and they do not jointly control temporal sharing in the shared stage and spatial sharing among co-running stages. This paper presents HorizonServe, a single-GPU omni-model serving system that coordinates request admission and GPU allocation under heterogeneous SLOs. HorizonServe profiles per-class first-response latency, protects requests with limited slack, rotates shared-stage opportunities across execution paths, and throttles the shared-stage streaming multiprocessor (SM) allocation when downstream stages are active. Across three omni-model workloads and two GPU platforms, HorizonServe improves SLO attainment by up to 4.9$\times$ in arrival-rate sweeps and 7.0$\times$ under downstream-heavy traffic, and reduces per-class first-response latency by 38.4--63.7\%.

Problem

• omni-model serving
• request scheduling
• GPU sharing
• heterogeneous SLOs
• first-response latency

Innovation

• omni-model serving
• GPU sharing
• request scheduling
• heterogeneous SLO
• first-response latency