Personalizing Large Language Model Agents with Small Policy Models

2026-07-31

AI Summary

This work addresses the challenge that large language model agents struggle to dynamically adapt their execution strategies—such as memory retrieval and tool usage—based on individual user feedback, while full-model fine-tuning remains costly and inapplicable to black-box systems. To overcome this, the authors propose FABLE, a method that introduces a lightweight policy layer external to the frozen model, factorizing execution decisions into memory access, information gathering, and response generation. By integrating feasible-set filtering with Bayesian contextual Thompson sampling, FABLE enables efficient online personalization using only scalar feedback, with theoretical regret bounds. Experiments demonstrate that FABLE significantly improves preference-sensitive behaviors in personalized reasoning, controllable feedback, and tool-use tasks, while maintaining competitive end-to-end task performance.

Abstract

Large language model (LLM) agents can retrieve memory, call tools, ask clarifying questions, and vary response style, yet adapting these execution decisions to an individual user remains difficult. Fine-tuning a separate LLM is costly or impossible for proprietary systems, while prompts and memory primarily expose user information to the agent rather than adapt its execution decisions from feedback. We formulate personalization of a frozen agent as online learning of a per-user execution policy from scalar feedback observed only for the executed action. We propose FABLE (Factorized Adaptive Bandit Layer for Execution), a lightweight policy layer outside a potentially black-box host agent. FABLE factorizes memory, information-acquisition, and response decisions so feedback updates related choices; filters actions through an externally specified feasible set before exploration; and learns user-specific residual preferences relative to a fixed default-and-cost score via Bayesian contextual Thompson sampling. Under a linear residual-reward model, a calibrated variant inherits an expected-regret bound against the best feasible action. We also characterize preferences unidentifiable under persistent feasibility constraints and provide anytime-valid false-promotion control. Across personalized-reasoning, controlled-feedback, and executable tool-use evaluations, FABLE improves several preference-sensitive behaviors relative to rule-only control while remaining competitive on end-to-end task performance.

Problem

• personalization
• large language model agents
• execution policy
• online learning
• user feedback

Innovation

• personalization
• bandit learning
• execution policy
• residual preference
• feasibility constraints