SoK: On the Offensive Potential of AI

2024-12-24

AI Summary

This study systematically investigates security and privacy risks arising from the malicious use of AI, focusing on dual threat dimensions—human actors and AI systems themselves. Method: Integrating findings from 95 academic papers, 38 InfoSec conference presentations, 12 expert interviews, and 549 user survey responses, we develop the first unified technical assessment framework for offensive AI. Employing multi-source knowledge fusion, cross-domain evidential triangulation, structured risk categorization, and empirically grounded threat modeling, we identify novel attack paradigms overlooked by prior work. Contribution/Results: We propose a scalable governance roadmap encompassing detection, attribution, and defense alignment. Our framework fills a critical gap in systematic offensive AI research, establishing both theoretical foundations and actionable guidance for AI security governance and responsible deployment.

Technology Category

• Philosophy and Ethics of AI: Privacy & Security
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Multiagent Systems: Adversarial Agents

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Our society increasingly benefits from Artificial Intelligence (AI). Unfortunately, more and more evidence shows that AI is also used for offensive purposes. Prior works have revealed various examples of use cases in which the deployment of AI can lead to violation of security and privacy objectives. No extant work, however, has been able to draw a holistic picture of the offensive potential of AI. In this SoK paper we seek to lay the ground for a systematic analysis of the heterogeneous capabilities of offensive AI. In particular we (i) account for AI risks to both humans and systems while (ii) consolidating and distilling knowledge from academic literature, expert opinions, industrial venues, as well as laypeople -- all of which being valuable sources of information on offensive AI. To enable alignment of such diverse sources of knowledge, we devise a common set of criteria reflecting essential technological factors related to offensive AI. With the help of such criteria, we systematically analyze: 95 research papers; 38 InfoSec briefings (from, e.g., BlackHat); the responses of a user study (N=549) entailing individuals with diverse backgrounds and expertise; and the opinion of 12 experts. Our contributions not only reveal concerning ways (some of which overlooked by prior work) in which AI can be offensively used today, but also represent a foothold to address this threat in the years to come.

Problem

• Artificial Intelligence
• Security Threats
• Privacy Issues

Innovation

• Artificial Intelligence
• Malicious Applications
• Threat Mitigation