Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation

2026-07-21

AI Summary

Offline reinforcement learning is often hindered by insufficient data coverage, and existing preference-based querying methods suffer from instability in sample selection and inefficient integration of feedback. This work proposes an uncertainty-aware joint optimization framework that estimates action uncertainty via a Morse network, designs a conservative preference querying strategy, and incorporates an adaptive regularization mechanism inspired by Conservative Q-Learning (CQL) to dynamically adjust data-level constraints. The approach jointly optimizes query selection and policy learning without requiring additional environment interactions. Experimental results on the D4RL benchmark demonstrate that the proposed method achieves performance competitive with or superior to current state-of-the-art approaches across multiple tasks.

Abstract

Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset, but its performance is fundamentally limited by dataset coverage. Action preference queries leverage expert feedback without additional environment interaction, enabling policy improvement during offline training. However, existing methods still face two key challenges: selecting informative preference queries and effectively exploiting the collected feedback. Current approaches typically rely only on the distance between policy actions and dataset actions for query selection, while enforcing fixed constraints that keep the policy close to queried preferences. Such strategies often lead to unstable policy updates and integrate poorly with value regularization. To address these limitations, we propose Conservative Query and Adaptive Regularization under Uncertainty Estimation, a lightweight framework that jointly improves preference querying and preference exploitation. Specifically, we employ a Morse network to estimate the uncertainty of policy actions with respect to the offline dataset. Based on this uncertainty, we introduce a conservative query strategy that selectively queries actions near the dataset to preserve Bellman-update stability, together with an uncertainty-aware adaptive regularization scheme that dynamically adjusts data-level constraints during policy optimization. We integrate our framework with CQL and evaluate it extensively on the D4RL benchmark. Experimental results demonstrate superior or competitive performance across a wide range of tasks.

Problem

• offline reinforcement learning
• preference queries
• uncertainty estimation
• policy regularization
• dataset coverage

Innovation

• offline reinforcement learning
• preference query
• uncertainty estimation
• adaptive regularization
• conservative policy update