The Economics of p(doom): Scenarios of Existential Risk and Economic Growth in the Age of Transformative AI

2025-03-10

AI Summary

This paper examines the trade-off between existential risk and economic growth arising from transformative AI (TAI)—AI systems that fully automate labor and decisively surpass human capabilities. We develop the first dynamic general equilibrium model embedding extinction risk endogenously within an intertemporal social welfare framework, integrating risk-weighted welfare analysis, scenario-based probability assignment, and rigorous sensitivity analysis. Theoretically, we show that even vanishingly small annual extinction probabilities (e.g., 10⁻⁶) generate infinite discounted welfare losses, necessitating substantial safety investment—and potentially justifying a moratorium on TAI development. Empirically, our calibrated results indicate that current global AI safety investment is grossly insufficient; a welfare-maximizing society should prioritize existential risk mitigation. This study provides the first rigorous, welfare-economic quantification of AI safety policy—establishing a formal foundation for evidence-based governance of advanced AI systems.

Technology Category

Philosophy and Ethics of AI: Safety, Robustness & TrustworthinessHumans and AI: Learning Human Values and PreferencesGame Theory and Economic Paradigms: Adversarial Learning

Application Category

Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applicationsSecurity and Privacy: Security and privacy of machine learning and AI applicationsResponsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent advances in artificial intelligence (AI) have led to a diverse set of predictions about its long-term impact on humanity. A central focus is the potential emergence of transformative AI (TAI), eventually capable of outperforming humans in all economically valuable tasks and fully automating labor. Discussed scenarios range from human extinction after a misaligned TAI takes over ("AI doom") to unprecedented economic growth and abundance ("post-scarcity"). However, the probabilities and implications of these scenarios remain highly uncertain. Here, we organize the various scenarios and evaluate their associated existential risks and economic outcomes in terms of aggregate welfare. Our analysis shows that even low-probability catastrophic outcomes justify large investments in AI safety and alignment research. We find that the optimizing representative individual would rationally allocate substantial resources to mitigate extinction risk; in some cases, she would prefer not to develop TAI at all. This result highlights that current global efforts in AI safety and alignment research are vastly insufficient relative to the scale and urgency of existential risks posed by TAI. Our findings therefore underscore the need for stronger safeguards to balance the potential economic benefits of TAI with the prevention of irreversible harm. Addressing these risks is crucial for steering technological progress toward sustainable human prosperity.

Problem

• Evaluates existential risks and economic outcomes of transformative AI.
• Highlights insufficient global efforts in AI safety and alignment research.
• Emphasizes need for safeguards to balance economic benefits and risks.

Innovation

• Evaluates existential risks and economic outcomes
• Advocates substantial AI safety and alignment investments
• Highlights need for safeguards against irreversible harm