Statistical Convergence of Transformer Encoder-Accelerated Robust Reinforcement Learning

2026-09-20

AI Summary

研究通过结合变压器编码器和鲁棒强化学习算法解决大规模状态-动作空间中计算最优动作值函数的问题，利用自然语言提示和保形预测方法加速收敛并减少初始误差。

Abstract

Obtaining the optimal action-value function in Markov decision processes is computationally intensive in large state--action spaces. In this study, we present statistically rigorous convergence results for a robust reinforcement learning algorithm warm-started by a transformer-based action-value function prediction, where natural language prompts encode task specifications. Our framework adopts the R-contamination model to characterize uncertainty in the state transition kernel, and employs conformal prediction to certify convergence via trajectory-level nonconformity scores constructed from the contracting Bellman residual. The resulting conformal quantile bounds the gap between the running and optimal action-value functions simultaneously over all iterations, thereby yielding a pre-certified stopping rule that requires little knowledge of the true transition kernel. Numerical case studies on perturbed maze environments of varying size and contamination level confirm that the transformer-based warm start measurably reduces the initial error and accelerates convergence, while the proposed conformal bounds track the true error trajectory more tightly than existing guarantees.

Problem

• Markov decision processes
• action-value function
• large state-action spaces
• computational intensity

Innovation

• Transformer-based warm start
• R-contamination model
• conformal prediction
• Bellman residual