Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities

2026-05-11

AI Summary

This work proposes a self-supervised method that requires no human annotations to evaluate whether language models rely on genuine reasoning rather than superficial shortcuts when performing mathematical text continuation tasks. The approach introduces an auxiliary prediction string \( Z \) and employs a cross-model likelihood scoring mechanism to assess whether \( Z \) conveys meaningful information by comparing the likelihoods of conditional versus unconditional continuations. To further probe model behavior, the study incorporates rigorous controls—including contextual manipulation and adversarial fine-tuning—to detect whether models exploit prompt engineering to deceive the evaluator. Evaluated on 1,363 equation continuation tasks, GPT-5.5 demonstrates significantly stronger performance than baseline models and exhibits robustness against shortcut exploitation, whereas GPT-5.4 nano fails to do so, thereby validating both the efficacy and discriminative power of the proposed framework.

Abstract

We introduce an automatically generated benchmark for predicting hidden text in technical papers. A paper supplies visible context $X$ and a hidden continuation $Y$; the evaluated model writes an auxiliary forecast string $Z$, and a separate scorer assigns next-token probability to $Y$ both with and without conditioning on $Z$. This gives a label-free test of whether $Z$ transmits information about the continuation, compared against controls where $Z$ is recent context rather than a forecast. Our main testbed is equation-suffix prediction: the predictor sees context and the first part of a displayed equation, then forecasts the rest. The task mixes surface-level arXiv/TeX text modeling with reasoning-sensitive inference; the suffix is one of many roughly equivalent continuations, so the benchmark is read statistically rather than item-by-item. On 1363 equation continuations from 138 recent physics and mathematics papers, forecasts from GPT-5.5, Opus 4.7, and GPT-5.4 nano all improve clipped likelihood over the context control under both Qwen3-8B and Kimi K2.6 scorers, distinguishing model families and reasoning-effort settings without human labels. To emulate shortcuts where $Z$ further primes the scorer rather than making a useful forecast, we also fine-tune the scorer on context-only prompts and apply it to held-out papers as a stronger control. GPT-5.5 forecasts still beat this fine-tuned control; GPT-5.4 nano forecasts do not. Longer prose/TeX continuations show positive but noisier lift over controls, concentrated near the beginning of the target. These results support cross-model likelihood scoring as a static benchmark and as a setup for probing shortcut vulnerabilities before reinforcement learning or model-selection optimization is applied.

Problem

• likelihood scoring
• mathematical text continuation
• shortcut vulnerabilities
• self-supervised benchmark
• equation-suffix prediction

Innovation

• self-supervised benchmark
• likelihood scoring
• shortcut vulnerability
• equation continuation
• cross-model evaluation