🤖 AI Summary
This study addresses the lack of scientifically grounded evaluation methods for assessing author-style personalization in large language models (LLMs), a gap that renders conventional metrics inadequate for capturing true stylistic fidelity. To remedy this, the work introduces authorship verification theory into LLM evaluation and proposes an integrated framework combining the LUAR authorship verification model, a decoupled trait-matching LLM-based evaluator, and classical function-word stylometric analysis. Experiments on 1,000 generated texts from 50 authors reveal a significant “authorship gap”: all inference-time personalization methods score markedly lower (0.484–0.508) than the cross-author baseline (0.626). Moreover, near-zero correlations (|r| < 0.07) among the three evaluation dimensions demonstrate that theoretically ungrounded assessments risk drawing misleading conclusions. This work establishes a calibrated, theoretically informed benchmark with absolute interpretability for style personalization evaluation.
📝 Abstract
Stylistic personalization - making LLMs write in a specific individual's style, rather than merely adapting to task preferences - lacks evaluation grounded in authorship science. We show that grounding evaluation in authorship verification theory transforms what benchmarks can measure. Drawing on three measurement traditions - LUAR, a trained authorship verification model; an LLM-as-judge with decoupled trait matching; and classical function-word stylometrics - we evaluate four inference-time personalization methods across 50 authors and 1,000 generations. The theory-grounded metric, LUAR, provides what ad hoc alternatives cannot: calibrated baselines, with a human ceiling of 0.756 and a cross-author floor of 0.626, that give scores absolute meaning. All methods score below this floor, from 0.484 to 0.508, exposing an authorship gap invisible to uncalibrated metrics. The three metrics produce near-zero pairwise correlations, with absolute r less than 0.07, confirming that without theoretical grounding, metric choice determines conclusions: an LLM judge declares a clear winner while LUAR finds no meaningful differentiation. These findings demonstrate the theory-benchmark cycle in action: authorship theory exposes evaluation failures that ad hoc benchmarks miss.