AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions

2025-08-22

AI Summary

Existing programming contest benchmarks suffer from insufficient problem difficulty, inadequate coverage, and low-quality test cases, leading to severe evaluation bias in assessing large language models (LLMs). Method: We introduce CodeContest, the first high-fidelity benchmark tailored to elite competitions (e.g., IOI, ICPC), comprising 1,200+ manually curated and annotated challenging problems. We further propose a human-in-the-loop test case generation framework that integrates automated construction with expert validation to ensure functional completeness and boundary-case robustness. Contribution/Results: CodeContest substantially widens the performance gap between LLMs and top human competitors—achieving sub-15% average pass@1 accuracy across mainstream models. It systematically exposes fundamental limitations of LLMs in complex algorithmic reasoning, multi-step logical composition, and error recovery. By establishing a more rigorous and trustworthy evaluation standard, CodeContest advances research in code generation and reasoning.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present AetherCode, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning.

Problem

• Evaluating LLMs' coding abilities in premier programming competitions
• Addressing overstatement of model proficiency versus elite programmers
• Providing rigorous assessment with expert-validated comprehensive test suites

Innovation

• Premier programming competition problems benchmark
• Hybrid automated and human expert test suites
• Rigorous reliable assessment for LLM capabilities