Red Teaming Contemporary AI Models: Insights from Spanish and Basque Perspectives

2025-03-13

AI Summary

This study identifies systemic biases and safety risks in mainstream multilingual large language models (LLMs) when deployed in Spanish–Basque bilingual contexts. We conduct the first bilingual, collaborative red-teaming evaluation—targeting OpenAI’s o3-mini, DeepSeek R1, and Spain’s indigenous ALIA Salamandra—using a novel adversarial probing framework comprising 670 multi-turn dialogues and cross-lingual safety assessments. Results reveal significant vulnerabilities across all models, with unsafe or biased response rates ranging from 29.5% (o3-mini) to 50.6% (Salamandra). Our key contribution lies in systematically extending red-teaming methodologies to the Spanish–Basque bilingual setting—a previously unexplored domain—and establishing the first empirical credibility assessment for small-parameter LMs within multilingual public AI infrastructure. This work fills a critical gap in trustworthy AI evaluation for low-resource languages and provides both empirical evidence and methodological guidance for AI governance in linguistically marginalized communities.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Large-scale security measurements

Abstract

The battle for AI leadership is on, with OpenAI in the United States and DeepSeek in China as key contenders. In response to these global trends, the Spanish government has proposed ALIA, a public and transparent AI infrastructure incorporating small language models designed to support Spanish and co-official languages such as Basque. This paper presents the results of Red Teaming sessions, where ten participants applied their expertise and creativity to manually test three of the latest models from these initiatives$unicode{x2013}$OpenAI o3-mini, DeepSeek R1, and ALIA Salamandra$unicode{x2013}$focusing on biases and safety concerns. The results, based on 670 conversations, revealed vulnerabilities in all the models under test, with biased or unsafe responses ranging from 29.5% in o3-mini to 50.6% in Salamandra. These findings underscore the persistent challenges in developing reliable and trustworthy AI systems, particularly those intended to support Spanish and Basque languages.

Problem

• Evaluating biases and safety in AI models
• Testing OpenAI, DeepSeek, and ALIA models
• Addressing vulnerabilities in Spanish and Basque AI systems

Innovation

• Red Teaming tests AI model vulnerabilities
• ALIA supports Spanish and Basque languages
• Manual testing reveals biases and safety issues