You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference

2026-09-29

AI Summary

This study addresses the uncontrollable quality, latency, and availability arising from price-only provider selection in open-source LLM inference. To this end, it proposes FACET, a market-aware routing framework that decouples provider selection for identical models into independent decision dimensions. FACET performs online learning grounded in multi-armed bandit theory and real-time endpoint measurements, while introducing task-level feasibility certification and an anchor-based fail-safe fallback mechanism. The research reveals a decoupling between market pricing and service quality. Validated through real-world traffic migration, the proposed approach demonstrates significant cost savings at equivalent quality levels while effectively circumventing suboptimal endpoints.

Abstract

Existing LLM routers choose among models using static per-model costs. We show that open-weight inference markets introduce a second, largely ignored decision axis: after choosing a model, a client must still choose which provider serves it. Measuring live endpoints across [nummodels] open models, competing providers, multiple task types, and three measurement waves, we find that provider choice cannot be inferred from the price list. The same model can vary sharply in quality, latency, availability, and price across providers; higher-priced providers are consistently faster, but price does not reliably predict quality or availability; and provider feasibility is task-selective, with one deployment nearly normal on knowledge tasks but catastrophically degraded on multi-step reasoning. We formulate same-model provider selection as a price-taker market-aware routing problem. A simple measured-map policy routes to the cheapest provider that is both quality-equivalent and healthy, yielding matched-quality savings while avoiding degraded endpoints. Because the map drifts, we introduce FACET, an online provider router that certifies per-(provider x task) feasibility facets and fails safe to an anchor before serving uncertified arms. Across relaxed deployment assumptions, FACET tolerates imperfect task assignment and sparse feedback, while systematic evaluator bias exposes a quality-signal trust boundary that can be mitigated with ground-truth probes or audits. Live provider runs further confirm that certification can move real traffic from a premium anchor to a substantially cheaper certified endpoint. Our results suggest that market-aware LLM routing must measure not only which model to use, but also who serves it.

Problem

• LLM routing
• open-weight models
• provider selection
• inference market
• model serving

Innovation

• Market-Aware Routing
• Provider Selection
• FACET
• Open-Weight LLM
• Online Certification