CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions

2026-06-02

AI Summary

Existing benchmarks for mathematical reasoning struggle to capture the dynamic process of multi-agent collaboration in solving open-ended problems. To address this gap, this work introduces CrowdMath, a dataset comprising 164 expert-annotated discussion threads from the MIT PRIMES–AoPS CrowdMath project (2016–2025), which comprehensively trace collaborative trajectories from problem formulation to proof completion. Each post is annotated with its functional role in the reasoning process—such as partial progress, error identification, or proof finalization. Leveraging this dataset, we define a new evaluation task that assesses models’ understanding of collaborative reasoning through macro F1 score on role classification and accuracy in next-post prediction. Experiments reveal that while state-of-the-art large language models achieve 83–88% accuracy in next-post prediction, their performance on functional role classification remains markedly limited, with the best macro F1 score reaching only 0.42, highlighting a critical deficiency in modeling collaborative mathematical reasoning.

Abstract

Large language models have made substantial progress on mathematical reasoning, but existing benchmarks typically evaluate well-specified problems with final answers, step-by-step solutions, or complete proofs. They do not capture collaborative open-problem solving: a setting in which participants propose partial arguments, identify gaps or errors in prior steps, repair flawed reasoning, and gradually synthesize incremental contributions into a proof. We introduce CrowdMath, a dataset of 164 expert-annotated progress chains from the MIT PRIMES--Art of Problem Solving (AoPS) CrowdMath program (2016-2025), a collaborative research initiative whose discussions have led to peer-reviewed publications. Each chain traces a multi-participant forum discussion from an open-problem statement to a completed proof. Posts are labeled by their functional roles in the evolving solution process, including partial progress, proof completion, erroneous reasoning, and error identification. We define evaluation tasks and benchmark six frontier models. Models achieve 83-88% accuracy on next-post prediction, suggesting that they can follow the local flow of mathematical discussion. However, they struggle to identify the functional significance of individual contributions with the best model achieving only 0.42 macro-F1 on post-role classification. CrowdMath exposes a gap between solving well-specified mathematical problems and understanding collaborative mathematical progress as it unfolds.

Problem

• collaborative problem solving
• open-problem solving
• mathematical reasoning
• crowdsourced research
• proof synthesis

Innovation

• collaborative mathematical reasoning
• crowdsourced research
• functional role annotation
• open-problem solving
• mathematical discussion dataset