MUSE: Multi-Tenant Model Serving With Seamless Model Updates

2026-02-12

AI Summary

This work addresses the challenge of decision threshold invalidation caused by score distribution shifts during model updates in multi-tenant Model-as-a-Service (MaaS) environments, which hinders rapid model iteration. To resolve this, the authors propose MUSE, a novel framework featuring a two-stage score transformation mechanism that maps model outputs to a stable reference distribution. Integrated with dynamic intent-based routing, MUSE enables seamless model updates and efficient sharing across tenants while preserving strict isolation guarantees. The architecture supports highly available, low-latency deployment and has been successfully deployed at scale by Feedzai, processing over 55 billion events annually. This deployment reduces model rollout time from weeks to minutes, significantly decreasing both fraud losses and operational costs.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Multiagent Systems: Mechanism Design

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In binary classification systems, decision thresholds translate model scores into actions. Choosing suitable thresholds relies on the specific distribution of the underlying model scores but also on the specific business decisions of each client using that model. However, retraining models inevitably shifts score distributions, invalidating existing thresholds. In multi-tenant Score-as-a-Service environments, where decision boundaries reside in client-managed infrastructure, this creates a severe bottleneck: recalibration requires coordinating threshold updates across hundreds of clients, consuming excessive human hours and leading to model stagnation. We introduce MUSE, a model serving framework that enables seamless model updates by decoupling model scores from client decision boundaries. Designed for multi-tenancy, MUSE optimizes infrastructure re-use by sharing models via dynamic intent-based routing, combined with a two-level score transformation that maps model outputs to a stable, reference distribution. Deployed at scale by Feedzai, MUSE processes over a thousand events per second, and over 55 billion events in the last 12 months, across several dozens of tenants, while maintaining high-availability and low-latency guarantees. By reducing model lead time from weeks to minutes, MUSE promotes model resilience against shifting attacks, saving millions of dollars in fraud losses and operational costs.

Problem

• model serving
• multi-tenancy
• decision threshold
• score distribution shift
• model update

Innovation

• model serving
• multi-tenancy
• score calibration
• seamless model update
• decision threshold