Answer-Set-Programming-based Abstractions for Reinforcement Learning

2026-05-29

AI Summary

This work addresses the challenge of poor generalization in real-world reinforcement learning due to large state spaces by introducing a novel approach within the CARCASS framework. For the first time, it replaces traditional Prolog with fully declarative Answer Set Programming (ASP) to construct a first-order logic-based model of Markov Decision Processes, thereby enabling logical abstraction in relational reinforcement learning. This shift significantly enhances the expressiveness and flexibility of abstract modeling. Empirical evaluations in the Blocks World and Minigrid domains demonstrate that, when augmented with domain knowledge, ASP effectively supports the construction of meaningful abstractions for reinforcement learning, confirming both its feasibility and advantages over conventional methods.

Abstract

Reinforcement Learning (RL) enables autonomous agents to learn policies from experience, but realistic problems often involve enormous state spaces, making learning and generalisation challenging. Abstraction and approximation are therefore essential. Relational Reinforcement Learning (RRL) offers a way to reason about objects and their relations, and the CARCASS framework by Martijn van Otterlo demonstrates how logical representations can model Markov Decision Processes (MDPs) in first-order domains. Originally implemented in Prolog, CARCASS leverages domain knowledge to create powerful abstractions. We explore Answer-Set Programming (ASP), which is a rich and, contrary to Prolog, fully declarative modelling language, to realise CARCASS abstractions. We evaluate our ASP-based implementation in case studies of two domains, viz. Blocks World and Minigrid. Our results indicate that CARCASS with ASP provides a promising approach to constructing abstractions for RL, especially when domain knowledge is available.

Problem

• Reinforcement Learning
• State Space
• Abstraction
• Relational Reinforcement Learning
• Markov Decision Processes

Innovation

• Answer Set Programming
• Relational Reinforcement Learning
• CARCASS
• Abstraction
• Declarative Modeling