LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

2026-07-20

AI Summary

Traditional reinforcement learning relies on scalar rewards, which struggle to distinguish semantically similar yet qualitatively distinct responses in open-ended tasks and are susceptible to reward hacking. This work proposes an Experiential Learning framework that reimagines the LLM-as-a-Judge paradigm as LLM-as-a-Coach, leveraging high-bandwidth textual feedback to distill transferable experiential knowledge in lieu of sparse scalar rewards. By integrating context-aware experience distillation, a policy internalization mechanism, and a reinforcement learning architecture tailored for non-verifiable tasks, the approach consistently outperforms score-based baselines across diverse policy models and feedback sources. The method substantially enhances out-of-distribution generalization and mitigates reward manipulation, offering a robust alternative to conventional reward-driven learning in complex, open-domain settings.

Abstract

Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.

Problem

• non-verifiable tasks
• open-ended tasks
• scalar reward
• textual feedback
• reward hacking

Innovation

• Experiential Learning
• LLM-as-a-Coach
• context distillation
• non-verifiable tasks
• reward hacking