Pareto Q-Learning with Reward Machines

2026-06-17

AI Summary

This work addresses the challenge of efficiently learning Pareto-optimal policies in multi-objective reinforcement learning under non-Markovian environments. It introduces Reward Machines (RMs) into the multi-objective reinforcement learning framework for the first time, integrating them with Pareto Q-learning (PQL). By leveraging the automaton-based decomposition of reward structures offered by RMs, the method maintains a set of vector-valued Q-function estimates in the cross-product MDP to approximate the Pareto front. This approach significantly improves sample efficiency, overcoming the limitation of conventional QRM methods that cannot handle multi-objective optimization, and enables the synthesis of Pareto-optimal policies inaccessible to standard QRM. Experimental results demonstrate that the proposed method converges faster and achieves superior performance compared to a naive PQL baseline directly applied to the cross-product MDP.

Abstract

We present Pareto Q-Learning with Reward Machines (PQLRM), a multi-objective reinforcement learning algorithm for tasks whose reward structure is specified by a set of reward machines (RMs). PQLRM combines Pareto Q-Learning (PQL), which maintains sets of vector-valued Q-estimates to approximate the Pareto front, with enhancements from Q-Learning with Reward Machines (QRM), which exploits the factored automaton structure of the reward signal. This yields a multi-policy algorithm that remains sample-efficient under non-Markovian, RM-encoded rewards. Experimental trials show that PQLRM converges faster than a naive PQL baseline applied to the cross-product MDP and can synthesize Pareto-optimal policies that QRM cannot.

Problem

• multi-objective reinforcement learning
• Pareto optimality
• reward machines
• non-Markovian rewards

Innovation

• Pareto Q-Learning
• Reward Machines
• multi-objective reinforcement learning
• non-Markovian rewards
• Pareto-optimal policies