Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval

2026-06-09

AI Summary

This work addresses the susceptibility of multiple-choice question answering (MCQA) evaluation to surface-form wording effects, which conflate models’ familiarity with phrasing and their actual knowledge. To mitigate this issue, the authors propose ParaEval, a novel framework that systematically exposes form-based biases in MCQA for the first time and introduces a robust evaluation paradigm based on multi-variant paraphrase generation and log-likelihood–optimal aggregation. Evaluated across large language models ranging from 1B to 120B parameters, ParaEval significantly enhances assessment fidelity: under controlled conditions preserving identical underlying knowledge, it reduces spurious performance gaps—exceeding two points in standard evaluations—to less than one point, thereby markedly improving the accuracy and reliability of model evaluation.

Abstract

Multiple-choice (MCQA) benchmarks are the standard for evaluating pretrained large language models, but their reliance on log-likelihood scoring makes them unreliable. Specifically, standard scores are highly sensitive to the exact phrasing (surface form) of the answers, conflating a model's familiarity with a specific phrase with its actual capability. We demonstrate this flaw using a controlled testbed of 1B-8B models trained on the same knowledge. Despite having identical knowledge, standard metrics falsely report a performance gap of over 2 points. To solve this, we propose ParaEval, an evaluation framework that queries models using multiple paraphrases per answer option. By scoring each model based on its most favorable phrasing, ParaEval successfully reduces the false performance gap to below 1 point. We confirm that these evaluation artifacts, and the improvements from ParaEval, persist in frontier 70B and 120B open-source models. Ultimately, ParaEval provides a robust and efficient way to evaluate true underlying capability rather than surface-form familiarity.

Problem

• MCQA
• evaluation bias
• surface form sensitivity
• language model evaluation
• paraphrasing

Innovation

• MCQA
• surface form sensitivity
• ParaEval
• paraphrase-based evaluation
• language model evaluation