SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs

2025-06-05

AI Summary

Existing methods for multi-objective alignment of large language models (LLMs) rely on explicit identity information—such as demographic attributes or predefined preference labels—limiting scalability and privacy. Method: We propose an unsupervised, interaction-driven personalized reward modeling framework that automatically synthesizes fine-grained user personas from natural conversational history and generates persona-adapted reward modeling prompts. Our core innovation is the first causal-inference-validated persona induction mechanism, integrating interaction importance filtering with LLM-as-a-judge fine-tuning. Results: On Chatbot Arena, our personalized judge achieves a 4.4% improvement in accuracy; on PersonalRewardBench—a newly constructed real-user benchmark comprising 854 interaction samples—it attains state-of-the-art performance, marking the first end-to-end personalized reward modeling approach that requires no external identity annotations.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent calls for pluralistic alignment of Large Language Models (LLMs) encourage adapting models to diverse user preferences. However, most prior work on personalized reward models heavily rely on additional identity information, such as demographic details or a predefined set of preference categories. To this end, we introduce SynthesizeMe, an approach to inducing synthetic user personas from user interactions for personalized reward modeling. SynthesizeMe first generates and verifies reasoning to explain user preferences, then induces synthetic user personas from that reasoning, and finally filters to informative prior user interactions in order to build personalized prompts for a particular user. We show that using SynthesizeMe induced prompts improves personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining SynthesizeMe derived prompts with a reward model achieves top performance on PersonalRewardBench: a new curation of user-stratified interactions with chatbots collected from 854 users of Chatbot Arena and PRISM.

Problem

• Inducing synthetic user personas for personalized reward models
• Improving personalized LLM-as-a-judge accuracy with guided prompts
• Enhancing reward model performance on user-stratified interactions

Innovation

• Generates synthetic user personas from interactions
• Filters prior interactions for personalized prompts
• Improves personalized LLM-as-a-judge accuracy