More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models

2026-09-29

AI Summary

This study addresses the "ordinal scale utilization bias" in direct decision-making models for ordinal label prediction, wherein models fail to fully exploit the complete decision space. Although representative approaches such as JEV achieve high accuracy, they exhibit severe decision compression. We formally define this bias for the first time and, through multi-dataset evaluations and candidate position randomization experiments, demonstrate that it constitutes a learnable optimization deficiency rather than an inherent architectural limitation. To mitigate this issue, we propose BA-LoRA, a targeted post-training technique. Following fine-tuning with our method, the relative utilization of gold-standard labels improves substantially from 47% to 86%, effectively enhancing model fidelity to the underlying ordinal scale.

Abstract

Direct-decision models turn text into low-latency structured labels and scores, making them attractive for classification and automatic evaluation. Yet reliability requires more than accuracy: a model must also use the ordinal decision scale supplied by the user faithfully. We analyze JEV~1.13 and three open KEV models. Our investigation begins with ANLI, where JEV assigns 38.8\% of all predictions and 51.3\% of errors to Neutral despite 74.95\% accuracy, nearly balanced gold labels, and balanced candidate positions. Across 36 ordinal datasets, final decisions use only 67--76\% of the effective gold support, versus 87--102\% on four nominal tasks. Randomizing candidate order weakens but does not remove this compression. Holding items and source scores fixed while balancing gold support and positions, we refine scales from $K=2$ to $14$; utilization falls for every model and reaches 26--75\% at $K=14$, although candidate probabilities remain broad for most models. Targeted BA-LoRA post-training raises gold-relative utilization from roughly 47\% to 86\% on eight supervised scales at both KEV sizes, showing that the compression is learned and modifiable rather than an immutable architectural limit. We call this ordinal scale-utilization bias: decision-stage candidate-space compression distinct from accuracy, gold imbalance, fixed position, and candidate count alone. The code and data are available at https://github.com/Glax147/jev_ordinal_scale_bia

Problem

• direct-decision models
• ordinal-scale bias
• scale-utilization bias
• candidate-space compression
• automatic evaluation

Innovation

• Ordinal-scale bias
• Direct-decision models
• BA-LoRA fine-tuning
• Scale utilization
• Candidate-space compression