Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification

2026-08-28

AI Summary

研究通过引入线性方程基准评估大语言模型在数学验证中的鲁棒性，发现模型对非标准但正确的解题过程敏感，提出监督微调等方法提升鲁棒性。

Abstract

Large language models (LLMs) are increasingly used as graders, verifiers, and process auditors, but most mathematical evaluations still emphasize final-answer accuracy. This can obscure whether a model can verify a non-canonical but valid solution trace. We introduce a controlled linear-equation benchmark for evaluating LLMs in the evaluator role. Each instance asks the model to judge final-answer correctness, step-level trace correctness, and the first incorrect step. Our evaluation of state-of-the-art open LLMs reveals a significant robustness gap: models that accurately evaluate canonical solutions often fail when presented with perturbed but logically equivalent variants. Across GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning, base models perform well on canonical traces but degrade substantially on perturbed traces, especially for error localization. On valid perturbed traces, base-model false-rejection rates reach 75.6-85.3%, showing strong sensitivity to canonical solution form. Supervised fine-tuning, distillation, and test-time compute improve robustness in some settings, but gains are model dependent and can trade off against canonical performance. The results show that reliable process-level verification remains challenging, and evaluator robustness should be measured separately from solver accuracy, even in a simple algebraic domain with exact ground truth.

Problem

• Large language models
• mathematical verification
• robustness
• canonical solutions
• perturbed traces

Innovation

• step-level verification
• robustness evaluation
• large language models
• supervised fine-tuning
• distillation