SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations

2026-06-03

AI Summary

Existing approaches struggle to reliably evaluate large language models (LLMs) in dynamic, multi-domain conflict mediation due to scarce realistic scenarios, high evaluation noise, and insufficient coverage of diverse social cognitive perspectives. This work proposes SoCRATES, a benchmark that leverages an agent-driven pipeline to generate authentic mediation scenarios across eight domains and systematically probes LLMs’ active mediation capabilities along five social cognitive dimensions. It introduces a novel topic-localized automatic evaluation mechanism to suppress off-topic noise. The framework enables, for the first time, systematic assessment of LLM mediation performance across multiple domains and social cognitive variables, achieving a human-expert alignment correlation of 0.82—double that of baseline methods. Evaluations on eight state-of-the-art LLMs reveal that even the best model closes only about one-third of the consensus gap, with performance varying significantly under different social cognitive conditions.

Abstract

Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting emotions, intentions, and context. Existing testbeds rely on a few expert-authored domains, vary mainly strategic posture, and score every turn against every topic, introducing off-topic noise. We introduce SoCRATES, a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It constructs scenarios from real conflicts through an agentic pipeline across eight domains, probes five socio-cognitive adaptation axes (strategic posture, party composition, history length, emotional reactivity, and cultural identity), and scores each topic only on the turns that advance it via a topic-localized evaluator. The evaluator reaches 0.82 alignment with human experts, more than doubling a per-turn baseline. Benchmarking eight frontier LLMs, we find that even the strongest mediator closes only about a third of the unmediated consensus gap under diverse and realistic testbeds, with performance varying sharply by socio-cognitive axis, highlighting that progress lies in social adaptation to diverse conditions.

Problem

• LLM mediation
• automated evaluation
• socio-cognitive variations
• multi-domain scenarios
• proactive mediation

Innovation

• proactive LLM mediation
• socio-cognitive adaptation
• topic-localized evaluation
• multi-domain benchmark
• agentic scenario generation