Evaluating and Benchmarking the System One Model Jev

2026-09-29

AI Summary

This study evaluates the performance of the Jev model in information pipeline decision-making to validate the advantages of a non-generative System One paradigm for classification and reasoning tasks. Through zero-shot benchmarking across 37 datasets, combined with confidence interval analysis, we systematically compare Jev against open-source large language models such as Qwen and Gemma in terms of accuracy and probabilistic calibration. Results demonstrate that Jev significantly outperforms the baselines on most tasks, achieving both high accuracy and superior calibration while exhibiting strong robustness to option permutation. This work confirms the effectiveness of non-generative models in structured decision-making scenarios, offering a novel paradigm for efficient and reliable AI-driven decisions.

Abstract

Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed options, a position on a rubric, or the probability that a statement is true, with probabilities the vendor describes as calibrated. Such models target small decisions in information access pipelines, such as routing queries, checking grounding, moderating content, or rating against a rubric. We evaluate Jev (jev-1.13.0) zero-shot on 37 datasets spanning classification, routing, natural language inference, reading comprehension, commonsense reasoning, moderation, legal clause analysis and rubric scoring, with one frozen template per dataset and full evaluation splits: 346,009 requests for under USD 10. For reference, we score Qwen3.8-27B and Gemma-4-E4B on identical requests via their exact next-token probabilities over the options. Jev reaches 95-99% accuracy on IMDB, SST-2, HellaSwag and ARC and 86.7% on Belebele across 122 languages. It beats Qwen on 27 of 37 datasets, with none of Qwen's nine leads outside the bootstrap intervals, and Gemma on all 37. All three models degrade on low-resource languages, fine-grained or noisy labels, and rubric-based quality judgments. Jev's choice probabilities are well calibrated and support selective prediction. Binary probabilities rank well but are poorly placed relative to a fixed 0.5 threshold; thresholds tuned on training data raise micro-F1 on UNFAIR-ToS from 0.50 to 0.75. Jev answers MMLU's calculation-heavy questions more accurately than other MMLU questions (94% vs. 91%), whereas both open models, and all three on C-Eval, find them harder. Rotating the options leaves Jev's accuracy unchanged and withholding the question drops it to near chance, ruling out shallow memorization but not memorized question-answer pairs. We release the code, harness and all raw responses.

Problem

• System One model
• benchmarking
• zero-shot evaluation
• probability calibration
• selective prediction

Innovation

• System One model
• calibrated probabilities
• zero-shot benchmarking
• selective prediction
• shallow memorization