PrefReward: Learning User Preference Matrix for Personalized Text Generation

2026-07-23

AI Summary

This work addresses the limitations of existing large language models in personalized text generation, which typically rely on implicit representations of user preferences, resulting in poor interpretability and difficulty capturing long-range contextual dependencies. To overcome these challenges, the authors propose PrefReward, a novel framework that introduces a structured user preference matrix to explicitly model individual writing styles for the first time. The framework further incorporates a KL divergence–based reward function to guide preference-aware decoding during generation. Evaluated on the LongLaMP benchmark, PrefReward significantly outperforms both non-personalized and retrieval-based baselines, achieving state-of-the-art performance in both generation quality and interpretability of personalization.

Abstract

Large Language Models (LLMs) have demonstrated remarkable ability in generating personalized content by leveraging user histories and contextual cues. However, most existing personalization approaches rely on implicit representations within model parameters, making it difficult to interpret user-specific preferences or effectively handle long-context dependencies. To address these challenges, we propose PrefReward, a novel preference-aware generative framework that explicitly models user styles through a structured preference matrix and integrates it into the decoding process as a reward signal. PrefReward consists of two stages: (1) extracting a user-specific preference matrix that summarizes individual stylistic tendencies, and (2) using the matrix to guide generation via a KL-divergence-based reward function. Experiments on the LongLaMP dataset show that PrefReward outperforms non-personalized and retrieval-based baselines in both generation quality and personalization interpretability.

Problem

• personalized text generation
• user preference
• long-context dependencies
• interpretability
• preference modeling

Innovation

• preference matrix
• personalized text generation
• reward-guided decoding
• KL-divergence reward
• user modeling