FMplex: Model Virtualization for Serving Extensible Foundation Models

2026-06-08

AI Summary

Existing model serving systems deploy foundation models independently for each downstream task, resulting in redundant backbone loading, GPU memory waste, and missed opportunities for batching and cost sharing. This work proposes FMplex, a novel system that introduces foundation model virtualization, enabling a single physical backbone to be shared as multiple logically isolated virtual foundation models (vFMs) with support for task-specific customization, independent lifecycles, and strong isolation. FMplex further incorporates a batch-aware fair scheduler that integrates task weights with intra- and inter-task batching mechanisms to enable efficient cross-task batching. Experiments across seven foundation models (16 variants) and 92 tasks demonstrate that FMplex reduces latency by up to 80% compared to spatial partitioning and by 33.3% over best-effort colocation, while increasing the number of concurrently supported tasks on a cluster by up to sixfold.

Abstract

Foundation models (FMs) are increasingly used as backbones for downstream tasks across language, vision, time-series, and multimodal applications. Yet existing model-serving systems deploy each customized task as an independent model instance, thereby replicating heavyweight backbones, wasting accelerator memory, and losing opportunities to amortize batching and loading costs. This paper presents FMplex, a serving system that treats FM backbones as a virtualization substrate for deployment sharing. FMplex presents each task with a virtual foundation model (vFM), a logically private FM instance backed by a shared physical FM. This abstraction lets independently customized tasks share a backbone while preserving task-specific extensions, independent lifecycles, and task-level isolation. In addition, we propose a batch-aware fair-queueing scheduler that combines weighted task-level sharing with inter- and intra-task batching across colocated tasks. We implement a FMplex-based serving stack spanning task construction, sharing-aware deployment, and runtime execution. Across 7 FM backbones (16 variants) and 92 downstream tasks, FMplex reduces latency by up to 80% over spatial partitioning and 33.3% over best-effort co-location, while hosting up to 6x more tasks at cluster scale.

Problem

• foundation models
• model serving
• virtualization
• resource sharing
• task isolation

Innovation

• model virtualization
• foundation models
• shared backbone
• virtual foundation model
• batch-aware scheduling