SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation

2025-04-17

AI Summary

Recommendation system evaluation has long suffered from discrepancies between offline metrics and actual online user behavior, exacerbated by scarce real-user data and stringent privacy constraints—necessitating high-fidelity user behavior simulation. To address this, we propose SimUSER: the first LLM-based, multi-module collaborative user behavior simulation framework. It integrates personality modeling, dynamic memory augmentation, perceptual modeling, and cognition-inspired decision-making to jointly align both micro-level interaction sequences and macro-level behavioral distributions with real-world patterns. Unlike prior approaches, SimUSER enables end-to-end synthetic user generation driven entirely by LLMs and supports offline A/B testing to iteratively refine recommendation policies. Empirical evaluation demonstrates that recommendation strategies optimized using SimUSER yield significant online improvements: +12.3% in click-through rate (CTR) and +9.7% in average session duration.

Technology Category

Cognitive Modeling & Cognitive Systems: Simulating Human BehaviorSearch and Optimization: Sampling/Simulation-based SearchHumans and AI: Intelligent User Interfaces

Application Category

User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systemsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recommender systems play a central role in numerous real-life applications, yet evaluating their performance remains a significant challenge due to the gap between offline metrics and online behaviors. Given the scarcity and limits (e.g., privacy issues) of real user data, we introduce SimUSER, an agent framework that serves as believable and cost-effective human proxies. SimUSER first identifies self-consistent personas from historical data, enriching user profiles with unique backgrounds and personalities. Then, central to this evaluation are users equipped with persona, memory, perception, and brain modules, engaging in interactions with the recommender system. SimUSER exhibits closer alignment with genuine humans than prior work, both at micro and macro levels. Additionally, we conduct insightful experiments to explore the effects of thumbnails on click rates, the exposure effect, and the impact of reviews on user engagement. Finally, we refine recommender system parameters based on offline A/B test results, resulting in improved user engagement in the real world.

Problem

• Evaluating recommender systems' performance with realistic user behavior simulation
• Addressing scarcity and privacy issues of real user data in evaluation
• Improving user engagement through refined recommender system parameters

Innovation

• SimUSER framework simulates user behavior with LLMs
• Persona, memory, perception modules enhance believability
• Offline A/B testing refines recommender system parameters