Benchmarks in Leipzig

2026-06-04

AI Summary

This study systematically evaluates the reasoning capabilities of large language models (LLMs) on research-level mathematical problems. Collaborating with 49 mathematicians, the authors constructed a benchmark dataset comprising 100 challenging mathematical problems and conducted the first comprehensive, multi-stage evaluation of five state-of-the-art LLMs, incorporating diverse strategies such as single-attempt inference, repeated sampling, and reflective rethinking. The results demonstrate a substantial improvement in model performance, with the number of correctly solved problems increasing from 59 in initial attempts to 98 after applying advanced reasoning protocols, thereby significantly narrowing the gap between current LLMs and human experts. This work establishes the first systematic evaluation framework for LLMs on large-scale, research-grade mathematical tasks and reveals the rapid evolution of their mathematical reasoning abilities.

Abstract

Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of the work was done during the 3-day workshop *Benchmarks in Leipzig* with 35 participants at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany. We present the resulting collection of 100 questions. We evaluated these questions in three stages: a single attempt by five state-of-the-art LLMs, followed by a 20-runs-per-model evaluation with three of these models, and finally a 3-run attempt with two heavy-thinking models. After Stage 1, 41 questions remained completely unsolved; after Stage 2, this count dropped to 16; and we concluded Stage 3 with only 2 unsolved questions. This demonstrates that the mathematical reasoning capabilities of LLMs are becoming impressive.

Problem

• mathematical reasoning
• large language models
• research-level mathematics
• benchmark evaluation
• unsolved problems

Innovation

• mathematical reasoning
• large language models
• benchmark dataset
• multi-stage evaluation
• heavy-thinking models