It's not the Language Model, it's the Tool: Deterministic Mediation for Scientific Workflows

2026-05-13

AI Summary

This work addresses the reproducibility challenges in scientific analysis caused by language models directly generating code. The authors propose a typed intermediary paradigm: structured expert interviews are used to extract domain-specific analytical workflows, which are then encapsulated into deterministic tools orchestrated by a locally deployed language model rather than relying on direct code generation. This approach guarantees bit-for-bit identical outputs across repeated executions and supports integration with proprietary binary formats and laboratory instruments. In photoluminescence analysis tasks, commercial large language models produced inconsistent or failed results across four runs, whereas the proposed system delivered stable, reproducible outputs. The framework has been successfully deployed on two instruments for six months, receiving positive user feedback and reducing analysis time from weeks to minutes.

Abstract

Language models can produce convincing scientific analyses, but repeated generations on the same data do not guarantee the same result. A researcher may regenerate an identical query and receive a different fit, a different peak position or a different analysis procedure, without an obvious way to decide which output to trust. We propose typed mediation, a pattern in which the model orchestrates deterministic tools rather than generating analytical code. Each tool encodes one researcher's exact procedure for one instrument, ported through structured interviews. The model selects which tool to call and with what parameters. The tool produces the result. Regeneration does not change it. We evaluate this claim by running the same photoluminescence analysis on four platforms, including three commercial foundation models, four times each with the same prompt. The typed tool produces identical results across all runs. The commercial platforms either vary in numerical output and analytical methodology across runs, or fail to produce valid results on the task. We deploy this pattern on two instruments serving users over approximately six months, with very positive user feedback. Both cases are very challenging: they involve proprietary binary formats and per-seat licensed software, which force the tool to remain on local infrastructure alongside the data and the instrument it operates. We argue that deployment topology is not just a preference, but a structural requirement of scientific tool mediation. The result is a practical pattern for deploying language models in scientific workflows where reproducibility is mandatory, reducing analysis time from weeks to minutes while guaranteeing identical outputs across runs.

Problem

• reproducibility
• language models
• scientific workflows
• non-determinism
• analytical consistency

Innovation

• typed mediation
• deterministic tools
• scientific reproducibility
• language model orchestration
• local deployment topology