Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs

2026-06-07

AI Summary

This study addresses the absence of a reliable, independent evaluation framework for consumer-facing health large language models (LLMs), which obscures inconsistencies and sycophantic behaviors in real-world use. It systematically identifies five structural barriers impeding robust assessment and introduces a novel methodology combining literature-derived simulated user personas, multi-turn prompt engineering, adapted standardized attitude scales, and black-box interface testing to evaluate personalization bias and response consistency across conversational turns. Findings reveal that model outputs are highly susceptible to latent input signals, commonly lack version tracking and conversation reset mechanisms, and evade detection by current evaluation approaches—particularly regarding tonal bias and omission of critical information. The work concludes with targeted governance recommendations to mitigate these shortcomings.

Abstract

Background: Consumer-facing large language models are now a common source of health information, and they interpret and personalize responses rather than retrieve them. Whether their responses vary across users is a clinical, equity, and governance question, sharpened by evidence that sycophantic responses can alter judgment and increase trust. Objective: To evaluate response variation and sycophancy in consumer-facing health LLMs under conditions resembling ordinary patient use. Methods: We constructed simulated user profiles differing in geography, browsing context, expressed beliefs, and social determinants of health, drawing on literature linking social context to health attitudes. We adapted validated instruments, including the Vaccination Attitudes Examination scale and reproductive attitudes scales, into multi-turn prompts designed to elicit clinically meaningful variation across users. Results: The evaluation encountered five linked barriers. Factual prompts produced stable responses that masked sycophancy emerging over multi-turn conversation. Browser-based interfaces did not disclose which signals influence outputs and could not be reset to a clean baseline. Large-scale testing was restricted by terms of service, rate limits, and bot detection. Accuracy-based criteria could not capture tone, framing, or omission, and LLM-as-judge methods risked shared alignment bias. Models changed without traceable version identifiers, preventing reliable replication. Conclusions: No reliable independent evaluation framework yet exists for examining how consumer-facing health LLMs behave in ordinary use. Oversight requires disclosure of personalization signals, stable version identifiers, researcher safe harbor programs, and post-deployment monitoring of health-related outputs.

Problem

• health LLMs
• response variation
• sycophancy
• independent evaluation
• consumer-facing AI

Innovation

• health LLMs
• sycophancy
• response variation
• independent evaluation
• personalization signals