CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

2026-03-13

AI Summary

This work addresses the notable absence of a comprehensive evaluation framework for large language models (LLMs) tailored to European Portuguese (PT-PT), particularly with respect to cultural alignment and safety. To bridge this gap, we present the first open-source LLM leaderboard and benchmark specifically designed for PT-PT, introducing a multidimensional evaluation framework that encompasses linguistic proficiency, cultural appropriateness, and behavioral alignment. Leveraging the Hugging Face platform, we developed and integrated several novel, localized assessment tasks, culminating in a publicly accessible leaderboard (https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard). This initiative fills a critical void in the systematic evaluation of LLMs for this linguistic variant and enables transparent, community-driven comparison and continuous monitoring of model performance.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.

Problem

• European Portuguese
• Large Language Models
• leaderboard
• cultural alignment
• model safeguards

Innovation

• leaderboard
• European Portuguese
• model safeguards
• cultural alignment
• open LLM