Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming

2026-06-10

AI Summary

While large language models have demonstrated strong performance in programming competitions, their capacity to assist human learning—particularly in identifying and repairing code errors—lacks systematic evaluation. This work introduces UOJ-Bench, the first multitask benchmark based on real user submissions from the Universal Online Judge, encompassing code generation, adversarial hacking, and repair tasks, with native evaluation infrastructure enabling realistic assessment. By integrating error identification and repair into the evaluation framework for large models, this study highlights their potential in educational contexts. Experiments reveal that even the strongest model achieves less than 50% error detection accuracy under a single-shot setting; however, incorporating test-time expansion strategies boosts success rates above 90%. Furthermore, the analysis uncovers hidden bugs in over 5% of “full-score” submissions across approximately 30 problems.

Abstract

Despite strong performance in competitive programming, the role of Large Language Models (LLMs) in supporting human learning in the same setting remains largely unexplored. In this work, we introduce UOJ-Bench, a benchmark designed to evaluate not only the problem-solving ability of LLMs, but also their ability to identify errors in human-written code -- a crucial educational activity traditionally supported by running test cases over online judge systems. UOJ-Bench consists of three distinct tasks: code generation, code hacking, and code repair, all constructed from real-world code submissions on the Universal Online Judge (UOJ) and evaluated through UOJ's native judging infrastructure. Our results show that under one-shot evaluation, even the strongest models fail to identify errors in more than 50% of a set of submissions that have been found to be incorrect by UOJ users. While test-time scaling improves success rates to above 90%, the substantial computational costs incurred from model inference limit its practicality for large-scale deployment. Despite these limitations, we find that the best-performing models under test-time scaling can uncover errors in over 5% of full-score submissions across roughly 30 problems, suggesting that frontier LLMs can already provide complementary signals beyond standard judging systems.

Problem

• code generation
• code hacking
• code repair
• Large Language Models
• competitive programming

Innovation

• code generation
• code hacking
• code repair
• test-time scaling
• online judge