FIGS: Evaluating Multi-Turn Sycophancy Without Penalizing Empathy

2026-09-30

AI Summary

This study addresses the inability of existing large language model evaluations to capture sycophantic dynamics in multi-turn dialogues, where empathy is frequently misidentified as principled compromise. To this end, we propose a dual-axis evaluation framework that introduces a novel taxonomy distinguishing sycophancy from calibrated responses by decoupling factual integrity from appropriate supportiveness. Methodologically, we construct a ten-turn adaptive dialogue simulator to generate authentic interaction scenarios and incorporate an automated judge mechanism for precise quantitative assessment. We ultimately release a benchmark comprising 500 test scenarios, which systematically reveals the persistent trade-off models face between maintaining honesty and providing support. This work establishes a new paradigm for research on large language model alignment.

Abstract

Large language models frequently fail to balance staying truthful with being supportive. They often exhibit sycophancy in responses to users, agreeing with false claims, offering unwarranted flattery, and giving advice skewed toward users' expressed views. In reality, sycophancy rarely happens in a single exchange; it may emerge organically as users repeatedly insist or subtly steer the dialogue over time. Current evaluations, however, rely on rigid, single-turn tests or fixed scripts that fail to capture these natural dynamics. Furthermore, these benchmarks often mistake showing basic empathy for yielding, penalizing models for acknowledging a user's feeling. This view may drive future models to over-correct into cold, dismissive rigidity. To address this gap, we introduce FIGS (Factual Integrity and Grounded Support), a dual-axis evaluation framework built around extended, realistic dialogue. We use an adaptive 10-turn conversational simulator that dynamically challenges the target model, reflecting how users repeat requests, push back, or steer a conversation toward a preferred answer. To accurately evaluate these trajectories, we apply a taxonomy that strictly separates Sycophancy (whether the model holds firm to the truth and keeps its praise proportional) from Calibrated Validation (showing empathetic understanding of the user's feelings without overdoing it). We release our complete testing environment, including 500 diverse multi-turn scenarios and an automated judge. Our evaluation of leading models reveals a consistent trade-off: over the course of a sustained interaction, current systems either slowly drift to sycophancy or over-correct into robotic detachment. This demonstrates that balancing honesty with appropriate support throughout a natural conversation remains a critical, unsolved challenge.

Problem

• multi-turn sycophancy
• large language models
• empathy evaluation
• factual integrity
• dialogue dynamics

Innovation

• Multi-turn Sycophancy
• Adaptive Dialogue Simulator
• Dual-axis Evaluation Framework
• Calibrated Validation
• Large Language Models