What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

2026-08-05

AI Summary

This study demonstrates that benchmark scores of large language models are often mistakenly interpreted as solely reflecting model capabilities, when in fact they are significantly influenced by undisclosed inference backends. Through a comprehensive cross-factorial experiment involving three models, five inference frameworks, six benchmarks, and four generation modes—combined with variance decomposition analysis—the authors systematically quantify the impact of backend choices on model behavior. Their findings reveal that nearly 39% of performance variance stems from the inference backend, with substantial differences persisting even under deterministic decoding and becoming especially pronounced in factuality-oriented tasks. The work further uncovers the structured and model-dependent nature of backend effects, urging the research community to adopt standardized reporting of complete inference configurations.

Abstract

Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes) we investigate how different tools (wrappers/backend) influence benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39\%) a practitioner sees out-of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework's default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.

Problem

• LLM behavior
• inference backend
• benchmark scores
• generation configuration
• model evaluation

Innovation

• inference backend
• LLM benchmarking
• generation configuration
• deterministic decoding
• model evaluation