Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs

2025-09-18

AI Summary

In LLM evaluation on multiple-choice question answering (MCQA), the tokenization of the trailing space after the “Answer:” prompt—commonly overlooked—induces accuracy fluctuations of up to 11% and substantially alters model relative rankings, undermining the reliability of existing comparisons. Method: The authors systematically analyze how diverse tokenization strategies affect next-token probability decoding, focusing on the segmentation of the space preceding the answer letter (e.g., “A”). Contribution/Results: They find that merging the leading space with the answer token (e.g., encoding “ A” as a single token) improves confidence calibration and prediction consistency across multiple mainstream LLMs, yielding stable and statistically significant performance gains. This finding holds across diverse models and datasets. The study advocates formalizing tokenization conventions as a mandatory component of MCQA evaluation protocols to enhance transparency, reproducibility, and fairness in LLM benchmarking.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

When evaluating large language models (LLMs) with multiple-choice question answering (MCQA), it is common to end the prompt with the string "Answer:" to facilitate automated answer extraction via next-token probabilities. However, there is no consensus on how to tokenize the space following the colon, often overlooked as a trivial choice. In this paper, we uncover accuracy differences of up to 11% due to this (seemingly irrelevant) tokenization variation as well as reshuffled model rankings, raising concerns about the reliability of LLM comparisons in prior work. Surprisingly, we are able to recommend one specific strategy -- tokenizing the space together with the answer letter -- as we observe consistent and statistically significant performance improvements. Additionally, it improves model calibration, enhancing the reliability of the model's confidence estimates. Our findings underscore the importance of careful evaluation design and highlight the need for standardized, transparent evaluation protocols to ensure reliable and comparable results.

Problem

• Tokenization variations affect MCQA accuracy up to 11%
• Space tokenization impacts model ranking reliability
• Standardized evaluation protocols needed for LLM comparisons

Innovation

• Tokenizing space with answer letter improves accuracy
• Strategy enhances model calibration and confidence reliability
• Standardized transparent evaluation protocols ensure comparable results