Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding

2026-08-07

AI Summary

This work addresses the persistent limitations of large language models (LLMs) in understanding real-world concepts, particularly their insufficient grasp of core conceptual attributes. To this end, the paper introduces the first controllable evaluation benchmark focused on geospatial concepts—such as direction, distance, and topology—employing synthetic data–driven question-answering tasks to systematically assess LLMs across three dimensions: abstractness, compositionality, and grounding. The study reveals significant deficiencies in current models’ ability to acquire and compose structured conceptual knowledge, while also elucidating how model scale and architecture influence conceptual understanding. These findings offer critical insights for the future design of more cognitively capable language models.

Abstract

Understanding concepts is fundamental to generalization. Despite their impressive performance on a wide range of tasks, Large Language Models (LLMs) still struggle with genuine concept understanding. Prior work has evaluated conceptual understanding in LLMs using natural-language benchmarks or narrowly scoped synthetic tasks, but these settings often conflate multiple skills or lack precise control over the underlying concepts and their properties. To support controlled probing of concepts in LLMs, we design tests on their core properties: abstraction, compositionality, and groundness. We set up a concept-centric benchmark, targeting spatial concepts such as direction, distance, topology, and their compositions, and use question answering tasks serving as a proxy. We conduct extensive experiments across multiple LLM architectures and training regimes to analyze how model scale and design impact conceptual understanding. The results reveal clear limitations in current LLMs and provide insights into the factors shaping their ability to acquire and compose structured concepts. Our findings shed light on how concept-based LLMs can be redesigned for improved information access and knowledge management. The code will be available at https://github.com/rd20karim/concept-probing.

Problem

• concept understanding
• large language models
• geo-spatial concepts
• abstraction
• compositionality

Innovation

• concept probing
• abstraction
• compositionality
• grounding
• geo-spatial reasoning