Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning

2026-06-04

AI Summary

This work addresses the challenges of slow convergence, inefficient exploration, and poor generalization in reinforcement learning caused by sparse rewards and heterogeneous task sequences. The authors propose a novel training framework that integrates symbolic planning, linguistic priors, and uncertainty awareness: symbolic trajectories generated by A* are used to fine-tune BERT, epistemic uncertainty is estimated via Monte Carlo Dropout, and an entropy-driven mechanism dynamically fuses suggestions from a large language model (LLM) with a PPO policy. This approach uniquely combines symbolic planning, pretrained language knowledge, and uncertainty-guided control. Evaluated on the MiniGrid-UnlockPickup benchmark, it achieves over a 9% improvement in execution accuracy and demonstrates significantly higher sample efficiency and reward AUC compared to existing baselines.

Abstract

Sparse rewards and heterogeneous task sequences remain persistent challenges in Reinforcement Learning (RL), often resulting in slow convergence, weak generalization, and inefficient exploration. We propose Uncertainty-Aware LLM-Guided Policy Shaping (ULPS), a novel framework that integrates a calibrated Large Language Model (LLM) into the RL training loop to provide structured, uncertainty-modulated behavioral guidance. ULPS employs an A*-based oracle to synthesize optimal symbolic trajectories, which are used to fine-tune a BERT-based language model. During training, this model supplies action suggestions whose influence is conditioned on epistemic uncertainty estimated via Monte Carlo (MC) dropout. An entropy-based blending mechanism adaptively balances LLM guidance and the learned policy (via Proximal Policy Optimization, PPO), allowing the agent to prioritize reliable priors while preserving adaptability. We evaluate ULPS on the MiniGridUnlockPickup benchmark and observe consistent improvements in success rate, reward efficiency, and sample complexity over unguided, uncalibrated, and standard RL baselines. ULPS achieves more than 9% improvement in execution accuracy after fine-tuning, requires fewer environment interactions, and yields higher reward AUC. Our results demonstrate that integrating symbolic A* trajectories, pretrained language priors, and uncertainty-aware control offers a principled and effective approach to multi-task reinforcement learning in sparse-reward domains, with potential extensibility to partially observable and multi-agent settings.

Problem

• sparse rewards
• heterogeneous task sequences
• reinforcement learning
• inefficient exploration
• slow convergence

Innovation

• Uncertainty-Aware RL
• LLM-Guided Policy
• Sparse-Reward Reinforcement Learning
• Symbolic Trajectory Synthesis
• Epistemic Uncertainty