Pancasila-Dilemmas: Evaluating Large Language Models on Indonesian Human Value Dilemmas Grounded in Pancasila

2026-07-20

AI Summary

This study addresses the gap in value alignment evaluation of large language models (LLMs), which has predominantly reflected Western perspectives and overlooked Indonesian indigenous values, particularly Pancasila. The authors introduce the first benchmark dataset grounded in Pancasila’s five core principles—belief in one God, just and civilized humanity, national unity, democracy guided by wisdom, and social justice—comprising 1,834 moral dilemmas derived from Indonesian news sources. These items were curated by native speakers and validated through multi-annotator voting. Evaluating 50 prominent LLMs using Probability Matching Score (PMS) and Majority Vote Agreement Score (MVAS), the study reveals that all models score below 0.5 on PMS and below 0.72 on MVAS, with notably poor performance on religious and unity-related dimensions, underscoring a significant deficiency in their understanding of Indonesian cultural and ethical norms.

Abstract

The value alignment of large language models (LLMs) is crucial for ensuring responses align with human intention and value preferences. However, most evaluations of value alignment focus on Western or universal values, while assessments grounded in the value systems of specific countries remain scarce. In this paper, we introduce Pancasila-Dilemmas, an evaluation dataset of 1,834 questions derived from Indonesian news, classified by 5 values of Pancasila: Religion, Humanity, Unity, Democracy, and Social Justice. This dataset reflects daily life in Indonesia, making it suitable for measuring the value alignment of LLMs deployed for Indonesia. To ensure a more rigorous evaluation, we choose scenarios containing dilemmas. The dataset is proofread by native speakers and answered by 5 diverse Indonesian citizens. We evaluate 50 closed- and open-source LLMs on our dataset. Results reveal that all evaluated LLMs achieves less than 0.5 Probability Match Score (PMS) and 0.72 Max-Vote Agreement Score (MVAS). Compared by each values, LLMs mostly struggle in Religion and Unity dilemma cases. This highlights a significant gap in capturing Indonesian values. The dataset is publicly available at https://github.com/tjunlp-lab/Pancasila-Dilemmas.

Problem

• value alignment
• large language models
• Pancasila
• Indonesian values
• moral dilemmas

Innovation

• value alignment
• Pancasila
• cultural grounding
• moral dilemmas
• LLM evaluation