Roomie: Interference-Aware Colocation for Efficient Model Serving

2026-07-18

AI Summary

This work addresses performance interference in multi-model co-location scenarios, where overlapping kernel execution times lead to violations of service level objectives (SLOs). To mitigate this, the authors propose a decoupled interference modeling approach that separates offline profiling from online prediction. By constructing an analytical interference model based on kernel-level resource utilization, the method avoids timing distortions inherent in traditional profilers. Furthermore, a pairwise greedy heuristic algorithm is introduced to efficiently approximate kernel-level interference among multiple models in polynomial time, enabling low-overhead deployment decisions. Experimental evaluation on both cloud and edge platforms demonstrates that the proposed technique reduces SLO violation rates by up to 3× while maintaining comparable or superior throughput performance.

Abstract

As demand for DNN inference grows, GPU capacity is increasingly oversubscribed, forcing operators to colocate multiple models on the same device in both cloud and edge deployments. Whether colocation succeeds or violates SLOs depends on the temporal overlap of kernels from concurrently executing models -- an effect that existing serving systems either ignore or approximate using aggregate resource profiles that fail to capture temporal dynamics. This paper presents Roomie, a model serving orchestration architecture that predicts and avoids kernel-level interference between colocated DNNs. Roomie decouples offline kernel profiling from online interference prediction. It uses profiling only to extract per-kernel resource configurations, and predicts interference with an occupancy-based analytical model immune to profiler-induced timing distortion. A pairwise greedy heuristic then approximates multi-model interference in polynomial rather than exponential time, and an online placement algorithm then uses these estimates to assign each incoming model to the GPU that minimizes predicted slowdown. Our experimental evaluation compares Roomie against state-of-the-art solutions across both cloud-grade server clusters and embedded edge devices, demonstrating that Roomie reduces SLO violations (i.e., inference latency) by up to 3x, while maintaining comparable, and in many cases superior, goodput relative to existing approaches.

Problem

• model colocation
• kernel interference
• DNN inference
• SLO violation
• GPU resource contention

Innovation

• interference-aware colocation
• kernel-level interference prediction
• occupancy-based analytical model
• model serving orchestration
• SLO-aware scheduling