Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning

2026-08-03

AI Summary

This work addresses the low sample efficiency of flat reinforcement learning in sparse-reward, long-horizon tasks and the static nature of knowledge representation in conventional hierarchical approaches by proposing a neuro-symbolic hierarchical reinforcement learning framework. The framework introduces an incrementally updated knowledge (InK) representation that enables dynamic adaptation. At the high level, symbolic planning is performed via belief-augmented Monte Carlo tree search integrated with prior knowledge, while the low level learns goal-directed motor primitives through goal-conditioned reinforcement learning enhanced with reward shaping. Experimental results demonstrate that the proposed method significantly improves sample efficiency in navigation tasks, validating the effectiveness of the incremental knowledge mechanism in enhancing hierarchical reinforcement learning performance.

Abstract

(Flat) Reinforcement Learning (RL) agents face significant challenges in environments with sparse rewards that require long-horizon reasoning. A compelling approach to improve sample efficiency is to incorporate knowledge into learning and decision-making. In standard Hierarchical RL (HRL), knowledge is encoded in a fixed, non-updatable form, such as architectural choices, and remains unchanged throughout learning. With fixed HRL, reasoning with incremental knowledge learned during exploration is impractical before sufficient environmental knowledge is acquired, leading to poor sample efficiency. In this work, we propose neurosymbolic HRL with {\em Incremental Knowledge (InK)}: symbolic high-level components perform {\em symbolic planning} (e.g. using $D^*$) on an updatable representation of current InK, while low-level goal-conditioned neural modules learn motion primitives through experience using reward shaping. Experiments on navigation tasks demonstrate that incorporating InK substantially improves sample efficiency. Additionally, to perform {\em optimal} symbolic planning given {\em prior} knowledge about the world, we develop Belief World Tree Search. The code is available at https://github.com/CPS-research-group/ink_bwts.

Problem

• Hierarchical Reinforcement Learning
• Sample Efficiency
• Incremental Knowledge
• Sparse Rewards
• Long-horizon Reasoning

Innovation

• Neurosymbolic Reasoning
• Incremental Knowledge
• Hierarchical Reinforcement Learning
• Symbolic Planning
• Sample Efficiency