Training a Conditioned Video Game Agent on a VLM Annotated Dataset

2026-08-06

AI Summary

This work addresses the limitations of traditional reinforcement learning, which relies on sparse and opaque reward signals from game engines and requires extensive trial-and-error interaction with the environment. The authors propose a novel approach that leverages vision-language models (VLMs) to automatically annotate human-interpretable reward signals from gameplay video datasets. Using these semantically meaningful rewards, a conditional agent is trained via offline reinforcement learning to execute behaviors aligned with high-level instructions. This method represents the first application of VLMs to generate interpretable rewards, eliminating dependence on environment-provided rewards and dense online interaction. The approach enhances policy interpretability while simplifying the training pipeline. Empirical results demonstrate the feasibility of this paradigm and highlight current challenges and limitations.

Abstract

Reinforcement Learning (RL) is a powerful but far from easy-to-use technique for policy learning. In the specific case of video games, access to the game engine is required to get rewards for training (e.g. to collect rewards from the environment). Furthermore, the proper identification and weighting of the rewards generally requires a difficult trial-and-error approach. Lastly, rewards are often sparse and understanding how they eventually affect the learned policy is a non-trivial exercise. To ease these issues we propose annotating a video game dataset with Vision Language Models (VLMs) instructed to extract human defined rewards. We show that offline RL can then be used to train a conditioned agent that responds accordingly to the desired returns and we discuss the difficulties and limitations that emerged in our early experiments.

Problem

• Reinforcement Learning
• reward sparsity
• video games
• reward engineering
• offline RL

Innovation

• Vision Language Models
• Offline Reinforcement Learning
• Reward Annotation
• Conditioned Agent
• Video Game AI