VLMs for Videogame Data Annotation

2026-08-06

AI Summary

This work addresses the challenge that vision-language models (VLMs) struggle to accurately interpret synthetic video game scenes and generate reliable reward signals. To overcome this limitation, the authors propose a suite of VLM adaptation strategies tailored to the characteristics of game visuals, including prompt engineering, fusion of outputs from multiple models, and controlled sequential input processing. Through systematic experiments, they analyze the impact of input length, resolution, and batch processing on model performance, validating the effectiveness of their approach in racing and other game environments. The results demonstrate that the proposed strategy significantly improves both the accuracy and efficiency of reward annotation, while also uncovering inherent limitations of current VLMs in comprehending complex game scenarios.

Abstract

Vision Language Models (VLMs) and Artificial Intelligence (AI) agents have revolutionized how engineers approach complex problems in real-world applications. Their adoption in video games is on the other hand limited by the extreme variability of the synthetic scenarios and their poor compliance with real-world physics. Here we investigate the use of VLMs for annotating video game frame sequences with reward signals, a task with several potential applications including, among others, conditioned training and offline reinforcement learning. We show that VLMs often struggle to answer basic questions on racing video games (although we observed a similar behavior on other game genres) and discuss countermeasures such as VLM output mixing and prompt optimization. We also show how input sequence length, resolution, and question batching affect the annotation quality and its token consumption.

Problem

• Vision Language Models
• Video Game Annotation
• Reward Signal
• Synthetic Scenarios
• Offline Reinforcement Learning

Innovation

• Vision Language Models
• reward annotation
• prompt optimization
• output mixing
• offline reinforcement learning