CreAgent: Towards Long-Term Evaluation of Recommender System under Platform-Creator Information Asymmetry

2025-02-11

AI Summary

This work addresses evaluation distortion in long-term recommender system sustainability assessment, arising from information asymmetry between platforms and content creators. We propose the first multi-agent simulation framework that explicitly models information asymmetry. Methodologically: (1) we design an LLM-driven creator agent integrating game-theoretic belief updating with dual-process cognition (System 1/System 2 thinking) for behavior modeling; (2) we develop a PPO-fine-tuned strategic behavior simulation paradigm enabling creators to adaptively respond to platform policies. Our contributions include significantly improved fidelity in long-term evaluation and empirical evidence that fairness- and diversity-aware algorithms can jointly optimize outcomes for platforms, creators, and users. The open-sourced simulation platform has been widely adopted by the research community.

Technology Category

• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems
• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Ensuring the long-term sustainability of recommender systems (RS) emerges as a crucial issue. Traditional offline evaluation methods for RS typically focus on immediate user feedback, such as clicks, but they often neglect the long-term impact of content creators. On real-world content platforms, creators can strategically produce and upload new items based on user feedback and preference trends. While previous studies have attempted to model creator behavior, they often overlook the role of information asymmetry. This asymmetry arises because creators primarily have access to feedback on the items they produce, while platforms possess data on the entire spectrum of user feedback. Current RS simulators, however, fail to account for this asymmetry, leading to inaccurate long-term evaluations. To address this gap, we propose CreAgent, a Large Language Model (LLM)-empowered creator simulation agent. By incorporating game theory's belief mechanism and the fast-and-slow thinking framework, CreAgent effectively simulates creator behavior under conditions of information asymmetry. Additionally, we enhance CreAgent's simulation ability by fine-tuning it using Proximal Policy Optimization (PPO). Our credibility validation experiments show that CreAgent aligns well with the behaviors between real-world platform and creator, thus improving the reliability of long-term RS evaluations. Moreover, through the simulation of RS involving CreAgents, we can explore how fairness- and diversity-aware RS algorithms contribute to better long-term performance for various stakeholders. CreAgent and the simulation platform are publicly available at https://github.com/shawnye2000/CreAgent.

Problem

• Long-term sustainability of recommender systems
• Information asymmetry between platforms and creators
• Simulating creator behavior with LLM and game theory

Innovation

• LLM-empowered creator simulation
• Game theory belief mechanism
• Proximal Policy Optimization fine-tuning