Lessons From Red Teaming 100 Generative AI Products

2025-01-13

AI Summary

This paper addresses the disconnect between theoretical benchmarks and real-world risks in generative AI—particularly large language model (LLM)—security evaluation. Drawing on red-teaming exercises across 100 Microsoft AI products, it proposes an ontology-based threat modeling–driven AI red-teaming methodology. The study establishes that AI red-teaming is not a replacement for conventional security assessment but rather a human-AI collaborative governance paradigm focused on dynamically evolving threats; LLMs both exacerbate existing vulnerabilities and introduce novel attack surfaces. The work distills eight reusable, empirically grounded practice principles into a standardized operational framework. This methodology has directly informed the secure iterative development of multiple Microsoft AI products and constitutes the first large-scale, empirically validated AI red-teaming framework for both academia and industry.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned: 1. Understand what the system can do and where it is applied 2. You don't have to compute gradients to break an AI system 3. AI red teaming is not safety benchmarking 4. Automation can help cover more of the risk landscape 5. The human element of AI red teaming is crucial 6. Responsible AI harms are pervasive but difficult to measure 7. LLMs amplify existing security risks and introduce new ones 8. The work of securing AI systems will never be complete By sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider.

Problem

• Generative AI Safety
• Large Language Model Assessment
• AI Red Teaming

Innovation

• Generative AI Security
• Red Team Testing
• Large Language Model Challenges