Analysis of Robustness of a Large Game Corpus

2025-04-04

AI Summary

This work addresses the lack of robustness and data sparsity in procedural content generation via machine learning (PCGML) for 2D tile-based game level generation. We formally define robustness as the sensitivity of level playability to small input perturbations—a first in PCGML—and propose a novel evaluation framework integrating local/global constraint modeling, adversarial perturbation detection, and statistical quantification. Leveraging four classic games, we construct the largest open-source tile-based level corpus to date—several times larger than existing benchmarks—significantly mitigating data sparsity. Experiments reveal that game levels exhibit substantially lower robustness in playability compared to conventional ML datasets; our method effectively identifies structurally fragile regions, enabling more reliable generative model training and assessment. Key contributions include: (1) a formal, playability-centered definition of robustness; (2) a structure-aware evaluation dimension; (3) a large-scale, diverse benchmark dataset; and (4) an open-source toolchain for quantitative robustness analysis.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Procedural content generation via machine learning (PCGML) in games involves using machine learning techniques to create game content such as maps and levels. 2D tile-based game levels have consistently served as a standard dataset for PCGML because they are a simplified version of game levels while maintaining the specific constraints typical of games, such as being solvable. In this work, we highlight the unique characteristics of game levels, including their structured discrete data nature, the local and global constraints inherent in the games, and the sensitivity of the game levels to small changes in input. We define the robustness of data as a measure of sensitivity to small changes in input that cause a change in output, and we use this measure to analyze and compare these levels to state-of-the-art machine learning datasets, showcasing the subtle differences in their nature. We also constructed a large dataset from four games inspired by popular classic tile-based games that showcase these characteristics and address the challenge of sparse data in PCGML by providing a significantly larger dataset than those currently available.

Problem

• Analyzing robustness of 2D tile-based game levels
• Comparing game level datasets to ML benchmarks
• Addressing sparse data in PCGML with large dataset

Innovation

• PCGML for 2D tile-based game content
• Robustness metric for sensitivity analysis
• Large dataset from classic tile-based games