GeoRef: Referring Expressions in Geometry via Task Formulation, Synthetic Supervision, and Reinforced MLLM-based Solutions

2025-09-25

AI Summary

This work addresses the critical deficiency in natural language referring expression comprehension (REC) for geometric reasoning in AI. We formally introduce the geometric referring expression comprehension task, requiring models to precisely localize points, shapes, and spatial relations in geometric diagrams given textual queries. To support this task, we construct GeoRef—the first high-quality benchmark dataset—and generate large-scale synthetic data using a structured geometric formal language. Methodologically, we propose Groupwise Relative Policy Optimization with Reward Alignment (GRPO), a novel reinforcement learning framework that integrates context-aware verification and re-generation mechanisms to enhance cross-modal alignment. Experiments demonstrate that GRPO outperforms standard supervised fine-tuning; models trained on GeoRef achieve substantial gains on downstream geometric reasoning tasks. These results empirically validate that robust geometry–language alignment is essential for mathematical understanding.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

AI-driven geometric problem solving is a complex vision-language task that requires accurate diagram interpretation, mathematical reasoning, and robust cross-modal grounding. A foundational yet underexplored capability for this task is the ability to identify and interpret geometric elements based on natural language queries. To address this, we introduce the task of Referring Expression Comprehension (REC) for geometric problems, which evaluates whether models can localize points, shapes, and spatial relations in diagrams in response to textual prompts. We present GeoRef, a benchmark dataset constructed from existing geometric problem corpora, featuring diverse, high-quality annotations and queries. Due to the lack of annotated data for this task, we generate a large-scale synthetic training dataset using a structured geometric formal language, enabling broad coverage of geometric concepts and facilitating model adaptation. We explore two fine-tuning approaches: Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). Our results show that GRPO significantly outperforms SFT by better aligning model behavior with task-specific rewards. Furthermore, we propose a verify-and-regenerate mechanism that detects incorrect predictions and re-infers answers using contextual reasoning history, further boosting accuracy. Notably, even state-of-the-art Multimodal Large Language Models (MLLMs) struggle with this task, underscoring the necessity of explicitly evaluating and strengthening geometric grounding as a prerequisite for robust geometric problem solving. Moreover, models trained on GeoRef demonstrate measurable improvements on downstream geometric reasoning tasks, highlighting the broader value of REC as a foundation for multimodal mathematical understanding.

Problem

• Identifying geometric elements from natural language queries in diagrams
• Localizing points, shapes, and spatial relations using textual prompts
• Improving geometric grounding for robust multimodal mathematical reasoning

Innovation

• Synthetic dataset generation using geometric formal language
• Group Relative Policy Optimization for fine-tuning
• Verify-and-regenerate mechanism with contextual reasoning