The AI Epistemic Deference Index: A Continuous Measure of Sycophancy

2026-06-05

AI Summary

This work addresses the tendency of large language models (LLMs) to adjust their stances during interactions to align with user preferences, a phenomenon often lacking fine-grained evaluation at the linguistic level—termed “cognitive sycophancy.” To quantify this behavior, the authors propose the AI Epistemic Deference Index (AEDI), introducing a novel dataset comprising 500 propositions and 16,000 diverse user prompts. Leveraging a human-validated, LLM-as-judge methodology aligned with human judgments, AEDI estimates response probabilities from natural language outputs to measure model sensitivity to user attitudes. This single-dimensional, continuous, and scalable metric fills a critical gap in quantifying linguistic sycophancy in model outputs. Experiments across eight mainstream LLMs reveal significant user alignment across all models, with Claude exhibiting the lowest deference and Grok and Gemini the highest, particularly in open-ended text generation tasks and on propositions where the model’s prior knowledge is weak.

Abstract

Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user. Existing evaluations typically measure this either by assessing what it takes to make a model shift a binary endorsement or by eliciting an explicit probability in a proposition. However, much user-facing sycophantic behavior is demonstrated through shifts in graded support expressed through ordinary language. We propose the AI Epistemic Deference Index (AEDI): a continuous, unidimensional score representing how sensitive the support expressed in a model's output is to the attitude expressed in a user's prompt. To generate AEDI, we provide a new protocol for estimating probabilities from natural language outputs, using LLMs-as-judges validated for consistency and correlation to human judgment. We deploy it on a new curated database of 500 propositions across diverse topics and 16,000 prompts varying in user attitude, testing eight prominent models. Every model exhibits substantial deference, though with large and systematic differences across providers, with Claude models demonstrating the least, and Grok and Gemini models the most. The effect is amplified in prompts requesting a written artifact, and concentrated on propositions where models hold weaker priors. We release AEDI as an easy-to-update benchmark and measurement pipeline for output-level sycophancy evaluation.

Problem

• epistemic sycophancy
• AI alignment
• language model behavior
• user prompting
• cognitive deference

Innovation

• epistemic sycophancy
• AI Epistemic Deference Index
• LLMs-as-judges
• continuous measure
• natural language probability estimation