Which Model Is Actually Serving You? IRIS: Budgeted Black-Box Auditing of Model Substitution and Routing Dilution in LLM Gateways

2026-07-22

AI Summary

This work addresses the risk that commercial large language model (LLM) gateways may fail to deliver the promised models, either through full substitution or request dilution, for which existing auditing methods lack a general black-box approach relying solely on output text. The paper proposes IRIS, a framework that constructs content fingerprints by prompting models to generate random strings, enabling detection of model substitution, estimation of dilution ratios, and backend provenance—all from returned text alone. IRIS is the first method to achieve these capabilities under a purely textual black-box setting without requiring prior signals or fixed probes, leveraging adaptive query budget allocation and statistical hypothesis testing. Experiments show an AUROC of 0.99 on the Qwen3 series; on OpenRouter, it achieves 0.85 average power to detect ε=0.3 dilution with ratio estimation error ≤0.04; and in cross-provider audits, it correctly identifies 14 out of 15 model pairs, improving target hit rate from 73% to 87%.

Abstract

Commercial LLM gateways mediate access to hosted models, but the served backend may not match the advertised one: it may substitute a cheaper model on every request or route only a fraction $ε$ of requests to it. Prior black-box auditors often need a privileged signal (log-probabilities, token ranks, or reference samples) or a target-specific probe, fix the query budget in advance, and return a yes/no verdict. We present $\mathrm{IRIS}$, an audit that needs only the returned text: it asks endpoints to generate random numbers or strings, fingerprints the backend, and is the first to combine, in one text-only audit, detection of whole-stream substitution and fractional dilution, attribution of the served backend, routing-fraction ($ε$) estimation, and a query budget it sizes itself. A cheap pilot fits the exponential query-error decay and freezes that budget before any suspect query is issued. On an intra-family Qwen3 ladder $\mathrm{IRIS}$ verifies the backend at $0.99$ AUROC and sharpens attribution as queries accumulate; across a commercial OpenRouter library it catches $ε{=}0.3$ dilution on margin-qualified pairs at $0.85$ mean power ($0.017$ false-positive rate) and recovers $ε$ to within $0.04$ for enrolled diluents; and a live cross-provider audit flags $14$ of $15$ same-model provider pairs by genuine quantization and kernel deviations, corroborated on third-party MET traces. Against comparable black-box auditors, $\mathrm{IRIS}$ matches or beats detection on shared tasks, and adaptive allocation lifts the matched-budget target-hit rate from $73$% to $87$%. Further experiments cover adversarial gateways, knob identifiability, unseen diluents, and false-positive control.

Problem

• model substitution
• routing dilution
• LLM gateways
• black-box auditing
• backend verification

Innovation

• black-box auditing
• model substitution
• routing dilution
• LLM gateways
• adaptive query budgeting