ProDVI: Programmatic Dynamics Priors for Value Network Initialization

2026-08-06

AI Summary

This work addresses the sample inefficiency of deep reinforcement learning stemming from the absence of prior knowledge. The authors propose a novel approach that leverages large language models to generate executable Python functions describing environment dynamics, which are then used to synthesize transition data. This synthetic data enables pretraining the state-action encoder within an Actor-Critic architecture via an auxiliary dynamics prediction task, yielding a dynamics-aware representation for initializing online reinforcement learning. Notably, this method is the first to integrate procedural dynamics priors with large language models, achieving effective initialization without requiring real-world data, high-fidelity simulators, or meta-learning. Crucially, the generated programs serve only for pretraining and do not rely on their exact correctness. Experiments demonstrate substantial improvements in sample efficiency for model-free reinforcement learning across OpenAI Gym and DeepMind Control Suite benchmarks.

Abstract

Deep Reinforcement Learning (RL) is notoriously sample inefficient. One contributing factor is that RL agents are typically initialized from scratch, forcing them to acquire task-relevant knowledge through online interaction. Existing approaches obtain informative initializations through pre-collected datasets, high-fidelity simulators, or meta-learning over related tasks, but these prerequisites may be difficult to access or even unavailable. In this paper, we propose Programmatic Dynamics Priors for Value Network Initialization (ProDVI), a framework that leverages the commonsense and domain knowledge encoded in large language models to initialize RL agents without relying on these resources. Specifically, ProDVI prompts a code-generating language model to produce executable Python functions that encode coarse hypotheses about environment dynamics. These functions are then used to generate synthetic transitions. Based on these transitions, we construct an auxiliary dynamics prediction objective to pretrain the state-action encoder of the value network in an actor-critic framework. The learned representation provides dynamics-aware inductive biases before online RL begins. Importantly, the generated programs are used only for representation pretraining and are not required to faithfully simulate the target environment. While the generated programs may be inaccurate, their induced initialization can be corrected through online learning from real transitions and rewards. Experiments on OpenAI Gym and DeepMind Control Suite tasks show that ProDVI can effectively improve the sample efficiency of model-free RL algorithms.

Problem

• sample inefficiency
• value network initialization
• deep reinforcement learning
• domain knowledge
• online interaction

Innovation

• Programmatic Dynamics Priors
• Value Network Initialization
• Large Language Models
• Sample Efficiency
• Reinforcement Learning