Beyond Accuracy: A Geometric Stability Analysis of Large Language Models in Chess Evaluation

2025-12-16

AI Summary

Current LLM-based chess evaluation relies on accuracy alignment (e.g., against Stockfish), yet high accuracy fails to distinguish genuine geometric reasoning from memorization of common positions. This work introduces the “geometric stability” framework—the first benchmark that measures response consistency under invariant transformations (rotation, mirroring, color inversion) as a metric orthogonal to accuracy—and formally defines and quantifies the “accuracy–stability paradox.” We conduct systematic, multi-axis symmetric perturbations and illegal-state detection tests on 3,000 chess positions. Evaluating GPT-5.1, Claude Sonnet 4.5, Kimi K2 Turbo, and Gemini 2.5 Flash, we find: GPT-5.1 exhibits over 600% surge in rotational error; Claude and Kimi achieve optimal dual robustness (accuracy + stability); and Gemini 2.5 Flash attains a 96.0% illegal-state rejection rate. These results expose critical limitations in prevailing evaluation paradigms and establish geometric stability as an essential dimension for assessing spatial reasoning in chess LLMs.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The evaluation of Large Language Models (LLMs) in complex reasoning domains typically relies on performance alignment with ground-truth oracles. In the domain of chess, this standard manifests as accuracy benchmarks against strong engines like Stockfish. However, high scalar accuracy does not necessarily imply robust conceptual understanding. This paper argues that standard accuracy metrics fail to distinguish between genuine geometric reasoning and the superficial memorization of canonical board states. To address this gap, we propose a Geometric Stability Framework, a novel evaluation methodology that rigorously tests model consistency under invariant transformations-including board rotation, mirror symmetry, color inversion, and format conversion. We applied this framework to a comparative analysis of six state-of-the-art LLMs including GPT-5.1, Claude Sonnet 4.5, and Kimi K2 Turbo, utilizing a dataset of approximately 3,000 positions. Our results reveal a significant Accuracy-Stability Paradox. While models such as GPT-5.1 achieve near-optimal accuracy on standard positions, they exhibit catastrophic degradation under geometric perturbation, specifically in rotation tasks where error rates surge by over 600%. This disparity suggests a reliance on pattern matching over abstract spatial logic. Conversely, Claude Sonnet 4.5 and Kimi K2 Turbo demonstrate superior dual robustness, maintaining high consistency across all transformation axes. Furthermore, we analyze the trade-off between helpfulness and safety, identifying Gemini 2.5 Flash as the leader in illegal state rejection (96.0%). We conclude that geometric stability provides an orthogonal and essential metric for AI evaluation, offering a necessary proxy for disentangling reasoning capabilities from data contamination and overfitting in large-scale models.

Problem

• Evaluates LLMs' geometric reasoning stability in chess beyond accuracy metrics.
• Proposes a framework testing model consistency under board transformations like rotation.
• Reveals an accuracy-stability paradox showing reliance on pattern matching over logic.

Innovation

• Geometric Stability Framework tests model consistency under transformations
• Comparative analysis of six LLMs reveals Accuracy-Stability Paradox
• Geometric stability provides essential metric for AI evaluation