From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation

2026-08-11

AI Summary

This work addresses two critical limitations in offline recommendation evaluation: the reliance on handcrafted feature engineering, which hinders scalability, and the susceptibility of zero-shot large language models (LLMs) to bidirectional rationalization—where contradictory yet seemingly plausible explanations for the same item undermine the reliability of user engagement prediction. To overcome these challenges, the authors propose a sequential behavior alignment framework that fine-tunes an LLM and incorporates preference optimization based on pairs of factual and counterfactual reasoning traces. This approach trains a personalized critic capable of directly predicting engagement from raw interaction logs while generating interpretable reasoning paths. The method is the first to identify and mitigate bidirectional rationalization in LLM-based recommendation evaluation, achieving a 32.19% relative improvement in Macro-F1 over zero-shot baselines on real homepage logs, matching the performance of production-grade feature-engineered systems while providing human-readable justifications.

Abstract

Traditional offline recommendation evaluation relies heavily on complex, manually maintained feature pipelines that are difficult to scale. While Large Language Models (LLMs) offer a promising alternative by predicting user engagement directly from raw text logs, empirical analysis in this study identifies a critical failure mode termed bidirectional rationalization. In a zero-shot setting, LLMs are found to convincingly argue for both positive and negative user engagement outcomes on the exact same item with identical evidence, highlighting the unreliability of off-the-shelf LLMs in predicting user engagement. To resolve this, we develop and apply a sequential behavioral alignment framework pairing fine-tuning with preference optimization over paired correct and counterfactual rationales. Evaluated on real-world homepage interaction logs, this aligned reasoning approach achieves a 32.19\% lift in Macro-F1 score over the zero-shot baseline and matches the production feature-engineered baseline. The results demonstrate that behavioral alignment mitigates bidirectional rationalization while delivering human-interpretable reasoning traces without manual pipeline overhead.

Problem

• bidirectional rationalization
• LLM reliability
• recommendation evaluation
• zero-shot prediction
• user engagement

Innovation

• behavioral alignment
• bidirectional rationalization
• LLM judges
• preference optimization
• recommendation evaluation