Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal standard

2026-08-02

AI Summary

Current evaluations lack precise ground-truth benchmarks for assessing the algorithmic reasoning capabilities of large language models. This work proposes the F-ICL benchmark, which leverages the Turing-complete machine F and its symmetric variant sF to enumerate all programs of length ≤13 and computes the Bayesian optimal posterior under the Levin–Solomonoff bounded universal prior, thereby providing the first computable theoretically optimal standard for evaluating in-context learning output distributions. The study introduces an innovative original-task/complement-task pairing to disentangle model inductive biases. Experiments across 37 open-source models (0.8B–675B parameters) and four leading proprietary systems show that peak accuracy reaches 92%, with 45 out of 46 models surpassing a keystroke-based baseline; however, performance exhibits no monotonic relationship with model scale or generation, and inductive biases do not consistently converge across model evolution.

Abstract

Whether large language models perform genuine algorithmic reasoning or mere pattern completion is hard to test, because most benchmarks lack a ground truth for correct inductive inference. We introduce F-ICL, an in-context-learning benchmark that supplies one exactly. Using the Turing-complete machine F, complement-symmetrised into sF to remove output-polarity bias, we exhaustively enumerate all 1.5 billion programs of length $L\le13$ and compute the Bayes-optimal posterior in closed form under a bounded universal (Levin--Solomonoff) prior; models are scored by how closely their served distributions approach it at matched evidence. Each task is paired with its bitwise complement, on which the optimum scores identically, so an original-twin gap isolates the model's inductive bias. Across 105 serving configurations spanning 37 open models (0.8B--675B) and frontier systems from four laboratories, models answer up to 92\% of queries correctly, yet 45 of 46 models yield distributions farther from the optimum than a keystroke reference, and their behaviour is bracketed by low-order prefix statistics fitted only on visible evidence. That reference is itself an algorithmic mixture, induced by a print-only machine with no loops, so the panel's implied measure sits closer to a loop-free mixture than to the loop-bearing optimum, independently of the reference machine. Updating is also non-monotone, which no prior explains: a Bayes-rational solved set can only grow in this realisable, noiseless setting, yet added examples produce $6{,}545$ solved-to-unsolved transitions against $13{,}702$ gains. The gap is not predicted by accuracy (Spearman $ρ=-0.19$, $p=0.21$), does not close with scale or across frontier generations in the serving modes that expose distributions, and is widened by instruction and reasoning post-training. F-ICL is released as an open, reproducible benchmark and toolkit.

Problem

• in-context learning
• algorithmic reasoning
• Bayes-optimal
• inductive bias
• language models

Innovation

• in-context learning
• Bayes-optimal reasoning
• algorithmic induction
• universal prior
• inductive bias