Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA

2026-07-28

AI Summary

This work proposes an automatic, annotation-free method for detecting geometric interpenetration anomalies in video games. By deploying autonomous exploration agents to collect gameplay frames and constructing an automated labeling pipeline to generate frame-level annotations, the study systematically evaluates the detection performance of six vision-language models—Gemini, GPT, Qwen, Gemma, Llama, and Ministral—in a zero-shot setting. It presents the first validation of VLMs’ capability to identify geometric interpenetration within a controlled game QA environment and provides an in-depth analysis of prompt design effects. Experimental results show that Gemini-3.1-Flash achieves the best overall performance and exhibits the highest robustness to prompt variations, whereas open-source models are more sensitive to prompting and are better suited as high-recall candidate filters.

Abstract

In this work, we study the use of Vision-Language Models (VLMs) for anomaly detection in an agent-driven game Quality Assurance (QA) pipeline focusing on geometry clipping. In this evaluation, a custom exploration agent navigates a game level to collect visual observations, while the automatic annotation pipeline provides frame-level clipping labels. This setup allows us to evaluate recent VLMs on a controlled anomaly detection task without manual annotation. We benchmark six recent VLMs (Gemini, GPT, Qwen, Gemma, Llama, and Ministral) under a zero-shot prompting setting and analyse their sensitivity to four prompt variants. Our results show that while the VLMs can capture visual cues associated with geometry clipping, they all produce substantial false positives on visually ambiguous frames such as near-contact geometry and partial occlusions. Gemini-3.1-Flash achieves the best overall accuracy and is the most robust to prompt variation, while open-source models exhibit large precision--recall swings depending on the prompt design. These findings suggest that current VLMs are best suited as high-recall candidate filters within multi-stage QA pipelines rather than as standalone bug detectors.

Problem

• geometry clipping
• anomaly detection
• video game QA
• autonomous agent
• Vision-Language Models

Innovation

• Vision-Language Models
• Geometry Clipping Detection
• Autonomous Game QA
• Zero-shot Prompting
• Anomaly Detection