Where Do Large Language Models Fail on Competitive Programming? A Taxonomy of Failures by Algorithm Type and Difficulty Rating

2026-06-02

AI Summary

This study addresses the limitations of aggregate pass rates in evaluating large language models on competitive programming by conducting a fine-grained failure analysis. We systematically assess GPT-4o and Claude Sonnet 4.6 on 315 Codeforces problems, introducing a balanced failure taxonomy spanning seven algorithmic categories and three difficulty levels. Through multi-dimensional attribution—including execution validation, ablation studies of zero-shot and chain-of-thought (CoT) prompting, and temperature control (T=0.2)—we uncover critical insights: CoT substantially reduces GPT-4o’s overall pass rate (from 46.0% to 36.8%), particularly degrading performance on greedy algorithms; while Claude demonstrates stronger logical reasoning under CoT (63.5% pass rate), it suffers a 244% increase in compilation errors. Over 70% of failures for both models stem from incorrect outputs, highlighting the pivotal influence of algorithm type and prompting strategy on model performance.

Abstract

Large language models (LLMs) demonstrate increasing proficiency on competitive programming benchmarks, yet technical reports predominantly publish aggregate pass rates, obscuring domain-specific vulnerabilities. We present a systematic empirical study of LLM failure patterns using a balanced taxonomy of 315 Codeforces problems across seven algorithm categories and three difficulty tiers. We evaluate GPT-4o and Claude Sonnet 4.6 under strict execution-based conditions, controlling for temperature (T = 0.2). To isolate the impact of reasoning frameworks on algorithmic correctness, we conduct an ablation study comparing direct zero-shot generation against zero-shot Chain-of-Thought (CoT). Our findings reveal a severe divergence from standard NLP benchmarks: forcing CoT aggressively penalizes GPT-4o, dropping its pass rate from 46.0% to 36.8% and exacerbating a critical weakness in Greedy logic. Conversely, while Claude maintains a higher logical baseline (63.5% under CoT), the expanded text generation severely degrades its markdown instruction adherence, causing its Compile Errors to more than triple (from 9 to 31, a 244% increase). Furthermore, failure-mode analysis indicates that Wrong Answer (WA) is the dominant verdict for both models--accounting for over 90% of GPT-4o's and roughly 70% of Claude's unaccepted solutions. These findings empirically demonstrate that standard prompt engineering techniques fail to bridge the algorithmic reasoning gap in competitive programming environments.

Problem

• competitive programming
• large language models
• algorithmic reasoning
• failure analysis
• Codeforces

Innovation

• competitive programming
• failure taxonomy
• Chain-of-Thought ablation
• algorithmic reasoning
• execution-based evaluation