BoN Appetit Team at LeWiDi-2025: Best-of-N Test-time Scaling Can Not Stomach Annotation Disagreements (Yet)

2025-10-14

AI Summary

This work investigates the applicability of test-time scaling (TTS) to natural language tasks characterized by substantial annotation disagreement (LeWiDi-2025), where ground-truth answers are inherently non-unique—contrasting sharply with verifiable domains like mathematics or programming. We systematically apply three canonical TTS strategies—model averaging, majority voting, and Best-of-N sampling—to this class of nondeterministic tasks for the first time. Empirical results show that model averaging and majority voting significantly improve both inter-annotator agreement and task performance, whereas Best-of-N yields no consistent gains, exposing its fundamental reliance on the existence of a single optimal output. This study fills a critical empirical gap in the literature on TTS under annotation ambiguity and provides methodological guidance and practical boundaries for reasoning optimization in subjective NLP tasks.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Calibration & Uncertainty Quantification
• Multiagent Systems: Agreement, Argumentation & Negotiation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Test-time scaling is a family of techniques to improve LLM outputs at inference time by performing extra computation. To the best of our knowledge, test-time scaling has been limited to domains with verifiably correct answers, like mathematics and coding. We transfer test-time scaling to the LeWiDi-2025 tasks to evaluate annotation disagreements. We experiment with three test-time scaling methods: two benchmark algorithms (Model Averaging and Majority Voting), and a Best-of-N sampling method. The two benchmark methods improve LLM performance consistently on the LeWiDi tasks, but the Best-of-N method does not. Our experiments suggest that the Best-of-N method does not currently transfer from mathematics to LeWiDi tasks, and we analyze potential reasons for this gap.

Problem

• Evaluating test-time scaling on annotation disagreement tasks
• Transferring Best-of-N method from mathematics to LeWiDi
• Analyzing performance gap in subjective annotation domains

Innovation

• Test-time scaling improves LLM outputs with extra computation
• Model averaging and majority voting boost performance on LeWiDi tasks
• Best-of-N sampling fails to handle annotation disagreements effectively