Beyond a Single Judge: Simulating Social Persona Panels for Generative UI Evaluation

2026-07-30

AI Summary

This study addresses the lack of efficient evaluation methods for Generative User Interfaces (GenUI) that reflect the diverse perceptions of real-world users. To bridge this gap, the authors propose the Evidence-Anchored, Socially Weighted Personality Panel (ESPP) approach, which introduces psychologically diverse personas synthesized via large language models. ESPP integrates a semantic-gated bounded-confidence opinion exchange mechanism with a Delphi-inspired social weighting aggregation scheme. This framework preserves individual ratings to reveal dimension-level disagreements among user subgroups while producing aggregate assessments that better align with human judgment. Experimental results demonstrate that ESPP significantly improves evaluation fidelity, increasing the Pearson correlation coefficient from 0.716 to 0.922—outperforming both single-rater and prompt-ensemble baselines—and uncovers substantial inter-dimensional disagreement even when overall user preferences appear consistent.

Abstract

Generative UI (GenUI) lets large language models synthesize a complete, renderable interface directly from a natural-language instruction, but evaluating the quality of what they generate remains an open problem. Human evaluation is costly and rater-variant, while LLM-as-a-judge is scalable but reflects only a single implicit viewpoint, unable to capture how different populations of real users actually perceive the same interface. We propose the Evidence-Grounded, Social-Weighted Persona Panel (ESPP), a three-stage GenUI evaluation method in which a panel of psychologically diverse, evidence-grounded personas independently rates a screenshot, exchanges opinions under a trait-derived, semantically-gated bounded-confidence mechanism, and is aggregated via Delphi-inspired social weighting into a single judgment. ESPP tracks human judgment substantially more closely than a naive single-pass judge, raising Pearson $r$ from $0.716$ to $0.922$, and a prompt-ensemble control recovers only about a third of this gap, isolating genuine persona and evidence grounding as the dominant source of improvement. Beyond this fidelity gain, retaining each panelist's individual rating further reveals that user subgroups agree on overall model rankings yet diverge sharply on specific rating dimensions, a structural disagreement a single homogeneous judge would systematically erase. The codes are available at https://github.com/Wuzheng02/ESPP.

Problem

• Generative UI
• Evaluation
• LLM-as-a-judge
• User perception
• Persona diversity

Innovation

• Generative UI evaluation
• Persona panel
• Evidence grounding
• Social weighting
• Bounded-confidence mechanism