rl latent tuning

Existing alignment methods—such as RLHF, human/internal feedback integration, test-time scaling, and diffusion guidance—lack a unified theoretical foundation, leading to instability (e.g., reward hacking, policy optimization divergence) and inflexibility in behavior control. Method: We establish the intrinsic equivalence among these paradigms, showing that soft-optimal N-sampling, resampling-based guidance, and reward modeling all instantiate implicit policy optimization. Building on this insight, we propose a novel resampling-based alignment framework that operates entirely at inference time: it dynamically reweights or resamples diffusion trajectories using heterogeneous feedback signals—explicit human ratings and implicit model self-assessments—without explicit RL training. Contribution/Results: Our approach eliminates reliance on unstable policy network updates and reward modeling pitfalls while preserving generation quality. It achieves superior controllability, robustness, and adaptability across diverse alignment objectives. Crucially, this work provides the first systematic theoretical unification of mainstream post-training alignment techniques under a coherent implicit optimization lens.

This work addresses the longstanding methodological, objective, and cultural divide between reinforcement learning and control theory by proposing a novel paradigm that integrates adaptive control with actor-critic reinforcement learning. The resulting framework enables data-driven optimization of controllers by unifying dynamic programming and online learning mechanisms, thereby reconciling modeling and optimization perspectives from both fields within classical motion control tasks. Theoretical analysis elucidates fundamental differences between the two approaches, while empirical results demonstrate the efficacy of the integrated strategy. This synthesis offers a solution for controlling systems with unknown dynamics that simultaneously guarantees stability and retains strong learning capabilities, fostering interoperability and synergistic development across disciplinary boundaries.

In reinforcement learning, adaptive interaction data—where the behavior policy is nonstationary—invalidates standard estimators, undermining asymptotic normality for off-policy counterfactual policy evaluation and dynamic treatment effect (DTE) inference. To address this, we propose a weighted Z-estimation framework that constructs time-varying adaptive weights to stabilize heteroskedasticity, achieving, for the first time in the RL off-policy setting, both consistent and asymptotically normal DTE estimation. Our approach integrates dynamic causal inference with asymptotic statistical theory, enabling rigorous hypothesis testing and construction of uniformly valid confidence regions. Simulation studies and real-world RL experiments demonstrate substantial improvements in confidence interval coverage and statistical power. The method provides the first solution for structural parameter inference under adaptive experimentation that simultaneously offers theoretical guarantees—namely consistency, asymptotic normality, and uniform validity—and empirical robustness.

This work addresses the limitations of traditional reinforcement learning, where static decoupling among the environment, policy, and reward model hinders adaptability to dynamic tasks—particularly in large language model (LLM) agent settings, where weak learning signals and poor generalization are prevalent. To overcome these challenges, we propose the first fully dynamic co-adaptive reinforcement learning framework that enables closed-loop joint optimization of the environment, policy, and reward model. Our approach integrates step-level and outcome-level feedback for policy training, employs consistency constraints to refine the reward model, and introduces a theory-driven mechanism for automatic environment adaptation. Extensive experiments on OSWorld, AlfWorld, and LiveBench demonstrate substantial performance gains: Qwen3-VL-8B-Thinking improves by 9.1%, while Qwen2.5-7B-Instruct achieves gains of 18.7% and 11.9%, respectively, validating the effectiveness and composability of our framework’s components.

Existing meta-learning methods suffer from limited generalizability, often being confined to specific algorithms or requiring differentiability assumptions. This paper proposes a general reinforcement learning–driven meta-learning framework that trains a teacher policy to dynamically guide arbitrary student algorithms—without imposing structural or differentiability constraints on the student. Key contributions include: (i) the first unified pedagogical paradigm for meta-learning; (ii) a parameter-behavior encoder that implicitly infers the student’s internal parameter state from its input-output behavior; and (iii) a reward function grounded in learning progress. Experiments across supervised and reinforcement learning tasks demonstrate that our framework significantly outperforms baselines relying on heuristic rewards and handcrafted state representations, validating its broad generalizability and empirical effectiveness.

This work addresses a key limitation in current reinforcement learning approaches for large language models, which predominantly rely on action-space exploration—such as temperature scaling—and struggle to effectively reorder tokens, often leading to training divergence or stagnation. To overcome this, the paper introduces Perturbed Parameter Policy Optimization (3PO), the first systematic framework leveraging parameter-space exploration. Built upon a variational formulation of the policy posterior, 3PO generates diverse trajectories through parameter perturbations and enhances exploration efficiency via a reward-based grouping mechanism. Evaluated on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks, 3PO consistently outperforms standard GRPO, yielding substantial gains in downstream performance with negligible computational overhead while significantly reducing zero-advantage groups and erroneous outputs.

Current reinforcement learning (RL) post-training of large language models (LLMs) is overly focused on policy gradient methods such as PPO and GRPO, largely neglecting the broader RL algorithmic landscape. This work proposes a modular analytical framework centered on three core dimensions—MDP formulation, exploration strategies, and learning mechanisms—and systematically maps classical RL techniques—including value functions, off-policy learning, bootstrapped credit assignment, intrinsic motivation, tree search, and curriculum learning—onto the LLM training context for the first time. The study reveals a predominant reliance in existing approaches on actor-only, Monte Carlo–style policy optimization and explicitly identifies underexplored yet promising directions, thereby offering a clear roadmap for future algorithmic innovation in LLM alignment and training.

This work addresses the lack of a unified theoretical framework for reinforcement learning, which has hindered systematic analysis of its convergence, sample complexity, and generalization. Building upon Markov decision processes and Bellman operators, the paper introduces a cohesive analytical framework that integrates tools from operator theory, stochastic approximation, convex duality, and function approximation. This framework encompasses a broad range of algorithms, including value iteration, policy iteration, temporal difference methods, off-policy learning, and constrained MDPs. By leveraging contraction mappings, monotone operators, martingale techniques, mirror/proximal optimization, concentration inequalities, and mixing process theory, the study establishes finite-sample performance bounds and asymptotic convergence guarantees for diverse reinforcement learning algorithms, thereby forging a rigorous theoretical bridge between probability theory, optimization, and statistics.

This work addresses the limited generalization of reinforcement learning policies under unmodeled or time-varying dynamics by proposing a trajectory-outcome-driven implicit dynamics representation that eschews reliance on predefined physical parameters. A task-specific smooth latent space is constructed via semi-supervised contrastive learning, and the authors theoretically establish a monotonic relationship between the regret bound in the target domain and the Lipschitz constant of the trajectory encoder. Leveraging this insight, they enforce Lipschitz constraints to optimize the geometry of the latent space, thereby enhancing robustness. Experiments on MuJoCo benchmarks demonstrate that the proposed method substantially outperforms parameter-centric baselines, effectively handling complex dynamics shifts while improving in-domain stability and interpretability of the latent representation.

Reinforcement learning agents often exhibit unpredictable goal-directed behaviors in out-of-distribution (OOD) environments, and the mechanisms underlying their generalization remain poorly understood. This work addresses this gap by adopting a developmental perspective, analyzing over one hundred sequential training curricula across more than 250 OOD environments to reveal the persistent influence of early-acquired goals on subsequent behavior. The authors propose a latent policy gradient method that leverages low-dimensional latent variables to effectively predict agent behaviors under unseen training curricula. This approach not only achieves high prediction accuracy and strong generalization capability but also offers interpretable insights into the mechanisms of goal generalization. Notably, it is the first to systematically uncover the structural regularities governing goal generalization in sequential training settings.