Momba: Network Modernization Improves Multi-Objective Reinforcement Learning

2026-08-07

AI Summary

This work addresses a critical limitation in multi-objective reinforcement learning: existing approaches typically employ simple feedforward networks with limited representational capacity, which hinders the quality of the solution set due to the complex trade-offs among objectives. To overcome this, the study systematically introduces modern neural network architectures into the field without altering the core algorithmic mechanisms. By incorporating observation and feature normalization, weight normalization, distributional return modeling, and entropy regularization, the proposed method substantially enhances function approximation capabilities. Empirical evaluations on multiple standard continuous-control benchmarks demonstrate that the approach significantly improves the quality of the generated Pareto front approximations, thereby filling an important research gap in neural architecture design for multi-objective reinforcement learning.

Abstract

Recent advances in deep reinforcement learning (RL) have shown that improving neural network architectures can yield substantial gains in sample efficiency and asymptotic performance without altering the underlying algorithms. In contrast, work on multi-objective reinforcement learning (MORL), which aims to discover a set of policies that balance trade-offs among conflicting objectives, has predominantly focused on algorithmic innovations, leaving the area of architectures underexplored. While the optimal policies and value functions can differ significantly depending on the trade-offs, MORL algorithms commonly represent them with simple feedforward networks conditioned on the trade-off. This raises the question of whether the performance of the algorithms could be improved with more expressive function approximators. In this paper, we integrate recent advances in neural network design: (i) observation and feature normalization, (ii) weight normalization, and (iii) modeling of distributional returns with an entropy-regularized MORL algorithm. The empirical results across standard continuous control benchmarks demonstrate that these changes substantially improve the quality of the produced solution sets without requiring major changes to the underlying algorithm.

Problem

• multi-objective reinforcement learning
• neural network architecture
• function approximation
• policy representation
• value function

Innovation

• multi-objective reinforcement learning
• neural network architecture
• distributional returns
• entropy regularization
• normalization techniques