WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

2026-08-04

AI Summary

Current evaluations of large language models (LLMs) predominantly rely on retrospective data, which are susceptible to memorization contamination and lack genuine prospective settings. This work proposes the first prospective evaluation framework free from information leakage, centered on the 2026 FIFA World Cup: prior to each match, six state-of-the-art LLMs—equipped with extended reasoning and native web-search capabilities—are tasked with real-time predictions of seven market outcomes, ensuring that questions are posed before ground-truth answers exist, thereby preserving evaluation integrity. We publicly release the complete prediction logs and scoring code, revealing an average model accuracy of 63.9%, comparable to betting-market favorites. However, despite high inter-model agreement, consensus does not guarantee correctness; majority voting proves ineffective, and overall performance remains highly homogeneous, with only marginal differentiation between top- and bottom-performing models.

Abstract

Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.

Problem

• forecasting
• large language models
• leakage-free evaluation
• prospective benchmark
• FIFA World Cup

Innovation

• prospective evaluation
• leakage-free benchmarking
• real-time forecasting
• frontier LLMs
• WorldCup Arena