What if automating AI R&D triggers an intelligence explosion?

2026-09-28

AI Summary

This study addresses the potential for AI code generation technologies to accelerate research and development automation, which may precipitate extreme loss-of-control risks such as an intelligence explosion—a phenomenon lacking systematic quantitative assessment in existing literature. To bridge this gap, this work proposes a novel quantitative analytical framework that integrates risk assessment models, policy simulations, and technological trend forecasting to systematically measure the accelerating effects of R&D automation on AI progress and evaluate potential extreme risk scenarios. The contributions of this research include elucidating the evolutionary pathways and societal implications of automation-driven intelligence explosions. Furthermore, it formulates targeted emergency governance strategies and monitoring mechanisms, thereby providing a scientific foundation and policy reference for mitigating AI loss-of-control risks.

Abstract

In contrast to even a year ago, AI systems now write most of the code inside the companies that build them. As more of the AI research and development (R&D) pipeline is automated, could AI progress radically accelerate in an "intelligence explosion," where years of advances are compressed into months or less? Preliminary evidence suggests that it could. In this work, we assess this evidence, analyze an intelligence explosion's potential impacts, and propose policy responses. AI systems are on track to automate most AI R\&D work within a few years, and possibly all of it. If this triggers an intelligence explosion, it could dramatically bring forward AI's benefits, but also pose extreme risks: capabilities growth could accelerate far beyond what society can keep up with, humanity could lose control over superhuman AI systems, and checks on power within and between states, companies, and branches of government could be severely eroded. Although there remains much uncertainty about these possibilities, the high stakes warrant serious further attention. Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion's impacts.

Problem

• intelligence explosion
• AI R&D automation
• existential risk
• superhuman AI
• AI governance

Innovation

• Intelligence Explosion
• AI R&D Automation
• Superhuman AI
• AI Governance
• AI Safety