RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning

2026-06-08

AI Summary

This work addresses a critical “evaluation gap” in large language models (LLMs) when assessing mathematical reasoning in authentic high school student solutions, revealing significantly weaker performance compared to synthetic data. The authors introduce RealMath-Eval, a novel annotated benchmark comprising 224 real student responses, which systematically demonstrates that human errors exhibit greater diversity and higher information-theoretic surprisal than structurally homogeneous synthetic errors. Through semantic embedding analysis, generative probability probing, and style-transfer experiments, the study shows that state-of-the-art models incur substantially higher mean squared error on real responses (≈2.96) than on synthetic ones (≈1.17). Crucially, superficial style transfer fails to bridge this gap, underscoring the models’ limited generalization to genuine human reasoning patterns.

Abstract

While Large Language Models (LLMs) have achieved near-perfect performance in \emph{solving} high-school mathematics, their ability to \emph{evaluate} the diverse reasoning processes of real human students remains under-examined. To bridge this gap, we introduce \textbf{RealMath-Eval}, a rigorously annotated benchmark of 224 real-world exam responses from high schools. Our initial evaluation reveals that even state-of-the-art LLM judges struggle significantly on this task, exhibiting a high Mean Squared Error ($\sim$2.96) against expert human grading. To probe a plausible explanation, we contrast this performance with a control setting where the same judges evaluate synthetic LLM-generated solutions. We identify a stark ``Evaluation Gap'': judges are considerably more accurate and consistent on synthetic text (MSE $\sim$1.17) but struggle to generalize to authentic student reasoning. Through semantic embedding analysis, we find that synthetic errors suffer from a ``structural collapse'' into predictable, low-dimensional linear subspaces, whereas human errors form a more diverse error space. Furthermore, generative probability probes suggest that human reasoning involves significantly higher information-theoretic surprisal, indicating that student reasoning transitions are more out-of-distribution for current models. Finally, we find that surface-level style transfer fails to close this gap. Our findings suggest that current LLM evaluation pipelines relying heavily on synthetic data may not adequately capture the diversity of authentic student mathematical reasoning.

Problem

• LLM evaluation
• human reasoning
• mathematical reasoning
• synthetic data
• evaluation gap

Innovation

• RealMath-Eval
• Evaluation Gap
• human reasoning
• synthetic data
• semantic embedding