🤖 AI Summary
This work addresses the challenge of evaluating model outputs in scenarios where ground-truth outcomes are delayed, censored, or private, rendering conventional code-based deterministic evaluation methods ineffective for immediate validation. The authors propose RouteCast, a novel framework that enables autonomous generation of auditable provisional prediction scores through typed, staged route modeling, reference-based analogy, and deterministic transformations, thereby supporting traceable and decomposable assessment of strategic pathways. Evaluated on 21 retrospective cases, RouteCast achieves an AUC of 0.756—significantly outperforming blind-evaluated large language models (AUC = 0.678) and performing comparably to identity-revealed LLMs (AUC = 0.761)—demonstrating its effectiveness and feasibility in settings with delayed ground truth.
📝 Abstract
Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast. RouteCast instantiates this regime for model-generated typed strategic routes: models propose candidate routes and structured factors; point-in-time evidence, reference classes, and deterministic transformations produce a provisional forecast-ranking; later outcomes evaluate the forecast. In a retrospective venture pilot on 21 binary-outcome cases (6 positive, 15 negative), the whole-packet RouteCast score showed preliminary retrospective discrimination (AUC 0.756, 95% CI [0.471,0.980]), while a blind LLM judge reached AUC 0.678 [0.419,0.897] and an identity-exposed LLM judge reached AUC 0.761 [0.515,0.944], consistent with recognition- or outcome-related leakage risk. A preregistered decomposition ablation on the same binary subset found that converting the identical inputs into typed staged routes was indistinguishable from the whole-packet score (Delta AUC = -0.144, 95% CI [-0.471,0.176]) and from a deterministic heuristic (Delta AUC = -0.089, 95% CI [-0.412,0.278]). The pilot establishes an auditable feasibility result and exposes failure modes; it does not establish prospective calibration, causal decision improvement, route-decomposition advantage, or cross-domain validity.