CRISP: Cultural Reward Modeling for Implicit Situated Propriety

2026-09-28

AI Summary

This study addresses the lack of cultural context awareness and behavioral appropriateness in large language models (LLMs) within open-ended social scenarios. We propose an implicit cultural norm modeling approach that constructs a cultural reward model combined with a norm-grounded supervision strategy to optimize cross-cultural interactions. Technically, we design a multi-agent collaborative data generation framework, establish the NormCompass evaluation benchmark, and employ the Group Relative Policy Optimization (GRPO) reinforcement learning algorithm alongside reward modeling for policy optimization. Experimental results demonstrate that our method significantly outperforms general-purpose baselines in both reward modeling and policy optimization, effectively enhancing the cultural sensitivity and social appropriateness of LLMs.

Abstract

As large language models (LLMs) are increasingly deployed across countries and regions, the ability to recognize and respond appropriately to diverse cultural contexts becomes increasingly important. However, existing research has largely focused on cultural knowledge or tasks with predefined response spaces, while open-ended culturally situated behavior remains comparatively underexplored. In this work, we introduce CRISP-RM, a culturally situated reward model that assigns rewards according to cultural appropriateness in open-ended social scenarios. During policy optimization, we further introduce Norm Grounding Supervision (NGS), providing guidance that enhances the policy's sensitivity to relevant cultural norms. To construct culturally situated data, we employ a collaborative multi-agent framework that instantiates implicit cultural norms into diverse social scenarios and further curate NormCompass as a dedicated testbed. We conduct comprehensive experiments to evaluate the effectiveness of CRISP-RM in both reward modeling and policy optimization. Best-of-\(N\) experiments show that CRISP-RM consistently outperforms strong general reward models. During GRPO policy optimization, CRISP-RM generally improves culturally situated behavior, while incorporating NGS yields further gains. Further analyses demonstrate the advantages of CRISP-RM in distinguishing culturally appropriate behavior beyond superficial fluency and politeness, while NGS provides complementary gains during policy optimization by improving norm grounding.

Problem

• Large Language Models
• Cultural Appropriateness
• Reward Modeling
• Open-ended Social Scenarios
• Policy Optimization

Innovation

• Cultural Reward Modeling
• Norm Grounding Supervision
• Multi-agent Framework
• Policy Optimization
• Open-ended Social Scenarios