GPT-5 at CTFs: Case Studies From Top-Tier Cybersecurity Events

2025-11-06

AI Summary

Evaluating the practical capabilities of frontier large language models (LLMs) in high-stakes, real-world cybersecurity competitions remains an open challenge. Method: This work presents the first systematic application of GPT-5 to elite Capture-The-Flag (CTF) competitions, integrating domain-adapted prompt engineering, chain-of-thought reasoning, and structured security knowledge to decompose and automate complex tasks—including vulnerability discovery, reverse engineering, and adversarial exploit development. Contribution/Results: Evaluated on the most difficult annual CTF event, the LLM-augmented team ranked 25th globally—outperforming 93% of participating teams—and achieved performance comparable to the world’s 3rd–7th strongest teams. The system successfully reconstructed complete solution paths for multiple high-difficulty challenges. This study not only demonstrates the feasibility and competitiveness of LLMs in professional security contests but also establishes a novel, AI-native paradigm for evaluating cybersecurity capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Large-scale security measurements

Abstract

OpenAI and DeepMind's AIs recently got gold at the IMO math olympiad and ICPC programming competition. We show frontier AI is similarly good at hacking by letting GPT-5 compete in elite CTF cybersecurity competitions. In one of this year's hardest events, it outperformed 93% of humans finishing 25th: between the world's #3-ranked team (24th place) and #7-ranked team (26th place). This report walks through our methodology, results, and their implications, and dives deep into 3 problems and solutions we found particularly interesting.

Problem

• Evaluating GPT-5's performance in elite cybersecurity CTF competitions
• Comparing AI capabilities with top-ranked human hacking teams
• Analyzing methodology and solutions for complex cybersecurity challenges

Innovation

• GPT-5 competes in elite cybersecurity CTF events
• AI outperforms 93% of human participants in hacking
• Methodology analyzes three specific cybersecurity problems