LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs

2026-08-05

AI Summary

Existing benchmarks for evaluating personalized large language models predominantly rely on textual personas or isolated behavioral signals, which inadequately capture personalization driven by everyday cross-domain behaviors. This work introduces the first evaluation benchmark grounded in long-term user interaction logs from daily-life contexts—such as clothing, food, housing, and transportation—employing a multi-stage, coarse-to-fine synthetic pipeline to generate data that closely mirrors real-world behavioral distributions while preserving privacy and scalability amid data sparsity. Experiments reveal that merely increasing context length or model scale fails to enhance personalization; directly retrieving fine-grained behavioral records outperforms compressed memory mechanisms, though strong personalization may entail privacy risks. This study pioneers cross-domain personalization evaluation based on general user behavior logs and highlights the critical role of behavioral evidence selection and cross-domain integration.

Abstract

Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals, providing limited evaluation of cross-domain behavioral personalization, where responses must be grounded in heterogeneous daily-life activities. To address this gap, we introduce LUNAR, the first benchmark for evaluating how LLMs personalize responses from longitudinal app interaction histories across universal daily-life domains, including clothing, food, housing, and mobility. To support scalable benchmark construction while mitigating data sparsity and privacy concerns, LUNAR uses a multi-stage coarse-to-fine synthesis pipeline grounded in real-world behavioral patterns. Fidelity analyses show closer alignment with real behavioral distributions than other synthetic benchmarks. Experiments on 19 mainstream LLMs show that access to behavioral logs is necessary but not sufficient for deep personalization: neither more context nor larger models guarantees better performance; effective personalization depends on selecting and integrating relevant evidence across domains. Direct retrieval of fine-grained behavioral records consistently outperforms compressed memory, while stronger personalization can come at the cost of privacy protection. These findings identify evidence selection, cross-domain integration, and privacy control as key challenges for personalized LLMs.

Problem

• personalized LLMs
• cross-domain behavioral personalization
• behavioral logs
• benchmarking
• privacy

Innovation

• personalized LLMs
• behavioral logs
• cross-domain integration
• synthetic benchmark
• privacy-personalization trade-off