offline rl evaluation

Real-world reinforcement learning faces significant challenges due to data scarcity and dynamically changing environments, leading to a growing gap between theoretical advances and practical deployment. This work proposes a practice-oriented, three-stage framework—comprising in-deployment online learning, inter-deployment offline analysis, and multi-round continual optimization—that systematically integrates recent advances in statistical reinforcement learning to enhance data utility, sample efficiency, and deployment strategies. By emphasizing the pivotal role of statistical methods in bridging the theory–practice divide, the framework offers both methodological guidance and novel research directions for developing reinforcement learning systems tailored to real-world scenarios.

Offline reinforcement learning (Offline RL) faces two key challenges: poor policy generalization and distributional shift—existing methods tend to overestimate out-of-distribution actions and require hyperparameter re-tuning for cross-task or cross-dataset transfer. To address this, we propose a dynamic policy switching mechanism at evaluation time, which adaptively fuses an offline RL policy with a behavior cloning policy based on dual uncertainty estimates: epistemic uncertainty (quantified via Monte Carlo Dropout) and aleatoric uncertainty (estimated via data density modeling). Our approach enables zero-shot cross-task transfer without parameter tuning and supports safe, zero-sample fine-tuning from offline to online settings. Evaluated on standard Offline RL benchmarks, it consistently outperforms state-of-the-art methods, achieving faster fine-tuning convergence and superior final performance.

This paper studies offline reinforcement learning in large-scale state spaces using only historical data, aiming to learn high-performance policies without environment interaction. Methodologically, it rigorously distinguishes between Bellman completeness and realizability—two distinct function approximation expressivity assumptions—and systematically characterizes data coverage conditions, including full-policy and single-policy coverage. It establishes a unified analytical framework linking coverage assumptions, algorithm design, and theoretical guarantees. Leveraging dynamic programming and statistical learning theory, the paper proposes several function-approximation-based offline RL algorithms and, for the first time under a unified perspective, derives tight sample and computational complexity bounds for them. The results precisely delineate the feasibility frontier of offline learning under various combinations of expressivity and coverage assumptions, thereby providing verifiable, theory-grounded guidance for algorithm selection and practical deployment.

To address inefficiency, instability, and the performance ceiling—where offline RL policies struggle to surpass behavioral cloning baselines—we propose a two-stage paradigm: supervised pretraining followed by constrained off-policy fine-tuning. First, policy and critic networks are jointly pre-trained via Monte Carlo value error minimization. Second, novel hybrid algorithms—including TD3+BC+CQL and EDAC+BC—simultaneously regularize both policy and value functions, enhancing behavior cloning stability and reliable policy improvement. Our approach reduces training time by over 50% on standard benchmarks while achieving markedly improved convergence stability. Crucially, under limited human demonstration data, it significantly boosts generalization and robustness. This work provides the first systematic empirical validation that supervised initialization yields dual gains in both performance and sample efficiency for offline reinforcement learning.

In online policy evaluation for reinforcement learning, outliers and heavy-tailed reward distributions severely degrade the accuracy of parameter and value function estimates. Method: We propose the first unified robust online statistical inference framework, introducing Bahadur-type expansions to temporal difference (TD) learning—enabling incrementally updated, asymptotically normal estimators—and integrating robust statistical estimation with online variance adaptation. Contribution/Results: Theoretically, we establish asymptotic normality of the estimator under both heavy-tailed rewards and adversarial contamination. Empirically, our method significantly improves estimation stability and confidence interval coverage in both synthetic and real-world RL tasks; it achieves over 40% higher robustness against interference compared to standard TD methods, providing a new paradigm for robust policy evaluation that is both theoretically grounded and computationally feasible.

Offline goal-conditioned reinforcement learning (GCRL) lacks standardized benchmarks, hindering systematic evaluation of algorithmic capabilities. To address this, we introduce OGBench—the first high-quality, capability-decoupled offline GCRL benchmark. It comprises eight environment categories and 85 diverse offline datasets, integrating both real-world and synthetic scenarios. We also open-source unified implementations of six representative algorithms, including behavior cloning, Q-learning variants, and inverse dynamics modeling. Crucially, OGBench is the first to orthogonally evaluate algorithms along four capability dimensions: behavior stitching, long-horizon reasoning, robustness to high-dimensional inputs, and handling of stochasticity. Empirical results reveal significant performance divergence across these dimensions, exposing critical limitations of existing methods. By providing a reproducible, quantifiable evaluation platform, OGBench fills a key gap in offline GCRL assessment and enables rigorous algorithm diagnosis, attribution analysis, and principled development of next-generation methods.

This work investigates how online versus offline data collection strategies affect the generalization and task performance of world models in model-based reinforcement learning. We identify that offline training suffers from insufficient state coverage, leading to out-of-distribution states at test time and substantial performance degradation. To mitigate this, we propose two mechanisms: (1) incorporating limited online interaction—under fixed or adaptive scheduling—to recalibrate the world model; and (2) augmenting offline datasets with exploratory trajectories to improve state coverage. Systematic evaluation across 31 continuous control benchmarks reveals that purely offline agents consistently underperform online baselines; however, even minimal online interaction restores—and often exceeds—their performance. Moreover, injecting exploration data significantly enhances the robustness and generalization of offline agents. Our study establishes a scalable co-design paradigm for data acquisition and model training, effectively balancing data efficiency with world model generalization.

In offline reinforcement learning, datasets exhibit heterogeneous quality and unknown provenance, necessitating a priori, training-free metrics for data quality assessment. This paper introduces the Bellman Wasserstein Distance (BWD), a value-aware distributional metric that quantifies dataset quality as the Wasserstein distance between Bellman error distributions induced by the behavior policy and a uniform random policy. BWD is constructed via state-conditioned optimal transport and a behavior critic, requiring neither environment interaction nor policy optimization. Evaluated on the D4RL MuJoCo benchmark, BWD exhibits strong correlation (mean Spearman’s ρ = 0.89) with oracle performance across diverse offline RL algorithms, accurately predicts policy returns, and—when incorporated as a regularization term—significantly improves learned policy performance. To our knowledge, this is the first training-free data quality evaluation framework grounded in optimal transport theory and Bellman consistency modeling.

This study addresses the poor out-of-distribution (OOD) generalization of reinforcement learning agents under changes in environment dynamics. To systematically evaluate rule-transfer generalization, the authors introduce a controllable benchmark based on one-dimensional cellular automata, where observation and action spaces remain fixed while only the transition rules are varied. The work proposes a standardized OOD evaluation protocol, statistical reporting guidelines, and an information-theoretic identity that characterizes the theoretical limits of uncertainty reduction. Empirical results demonstrate that algorithms exhibiting strong in-distribution performance suffer significant degradation under rule transfer, and that high-variance OOD evaluations require extensive repetition to ensure reliability. This work provides a reproducible evaluation framework alongside theoretical insights into generalization under dynamic rule shifts.

This work addresses the challenges in offline reinforcement learning where policies are prone to out-of-distribution (OOD) errors due to deviation from the training data distribution and struggle to balance expressive power with multimodal action modeling. To tackle these issues, the paper proposes ReFORM, a novel method that leverages a flow-based policy trained via behavioral cloning from a bounded source distribution to capture the action support set. ReFORM introduces, for the first time, reflected flows to generate bounded noise, which inherently respects the support constraints while optimizing policy performance—thereby circumventing the limitations imposed by conventional statistical-distance penalties on policy improvement. Evaluated on 40 tasks from the OGBench benchmark, ReFORM achieves state-of-the-art performance across the board using a single set of hyperparameters, significantly outperforming all baseline methods that require manual tuning.

This work addresses the challenge in offline reinforcement learning that existing metrics inadequately evaluate trajectory-level data removal, often conflating forgetting, residual memorization, and policy degradation. To this end, the paper introduces TOUR, a comprehensive benchmark that integrates trajectory partitioning, non-member controls, retraining references, performance anchors, and diverse privacy attacks—including action error, representation similarity, and query-limited attacks—to systematically assess forgetting quality. Empirical results demonstrate that single membership scores are unreliable; the efficacy of deletion methods is highly environment-dependent (e.g., D4RL versus AntMaze); retraining and fine-tuning generally outperform GA+Refit; and robust evaluation necessitates a holistic consideration of attack types, utility preservation, and architectural evidence.