SySLLM: Generating Synthesized Policy Summaries for Reinforcement Learning Agents Using Large Language Models

2025-03-13

AI Summary

Reinforcement learning (RL) policies, tightly coupled with neural networks and complex reward functions, exhibit unpredictable and opaque behavior, severely undermining human trust. Existing global policy summarization methods rely on limited state-action demonstrations and lack capacity for behavioral pattern abstraction, while their explanations remain highly subjective and user-dependent. Method: We propose the first LLM-based, synthesis-driven textual summarization paradigm for RL policies—integrating trajectory analysis, domain-aware world knowledge injection, and hallucination suppression to generate high-fidelity, structured, and semantically interpretable policy summaries. Contribution/Results: Experiments demonstrate that our method accurately reproduces expert domain insights in human evaluation, achieves significantly higher user preference over conventional demonstration-based summarization, and matches or exceeds baseline performance in agent identification tasks—establishing a new standard for trustworthy, human-aligned RL policy explanation.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Policies generated by Reinforcement Learning (RL) algorithms can be difficult to describe to users, as they result from the interplay between complex reward structures and neural network-based representations. This combination often leads to unpredictable behaviors, making policies challenging to analyze and posing significant obstacles to fostering human trust in real-world applications. Global policy summarization methods aim to describe agent behavior through a demonstration of actions in a subset of world-states. However, users can only watch a limited number of demonstrations, restricting their understanding of policies. Moreover, those methods overly rely on user interpretation, as they do not synthesize observations into coherent patterns. In this work, we present SySLLM (Synthesized Summary using LLMs), a novel method that employs synthesis summarization, utilizing large language models' (LLMs) extensive world knowledge and ability to capture patterns, to generate textual summaries of policies. Specifically, an expert evaluation demonstrates that the proposed approach generates summaries that capture the main insights generated by experts while not resulting in significant hallucinations. Additionally, a user study shows that SySLLM summaries are preferred over demonstration-based policy summaries and match or surpass their performance in objective agent identification tasks.

Problem

• Generates synthesized policy summaries for RL agents
• Improves human trust by explaining complex RL policies
• Uses LLMs to create coherent, interpretable policy descriptions

Innovation

• Uses large language models for policy summarization
• Generates textual summaries from complex RL policies
• Enhances user understanding and trust in RL applications