When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity

2025-10-20

AI Summary

Conventional evaluation relies heavily on scalar accuracy metrics, failing to characterize how models internally represent ambiguous samples—especially those exhibiting substantial human annotation disagreement. Method: This work pioneers the application of topological data analysis (TDA), specifically the Mapper algorithm, to the fine-tuned embedding space of RoBERTa-Large on the MD-Offense dataset, enabling systematic geometric characterization of ambiguity encoding. Unlike linear (e.g., PCA) or locally preserving (e.g., UMAP) dimensionality reduction methods, Mapper captures non-convex, modular decision regions in high-dimensional embeddings. Contribution/Results: We identify that over 98% of Mapper-generated connected components achieve ≥90% prediction purity and localize critical failure modes—including boundary collapse and overconfident clusters. Furthermore, we introduce the first topology-driven, connectivity-based quantitative metric, uncovering an implicit tension between “structurally high-confidence” representations and “label-level low-certainty.” This establishes a novel paradigm for probing ambiguity modeling mechanisms in large language models.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Language models are often evaluated with scalar metrics like accuracy, but such measures fail to capture how models internally represent ambiguity, especially when human annotators disagree. We propose a topological perspective to analyze how fine-tuned models encode ambiguity and more generally instances. Applied to RoBERTa-Large on the MD-Offense dataset, Mapper, a tool from topological data analysis, reveals that fine-tuning restructures embedding space into modular, non-convex regions aligned with model predictions, even for highly ambiguous cases. Over $98%$ of connected components exhibit $geq 90%$ prediction purity, yet alignment with ground-truth labels drops in ambiguous data, surfacing a hidden tension between structural confidence and label uncertainty. Unlike traditional tools such as PCA or UMAP, Mapper captures this geometry directly uncovering decision regions, boundary collapses, and overconfident clusters. Our findings position Mapper as a powerful diagnostic tool for understanding how models resolve ambiguity. Beyond visualization, it also enables topological metrics that may inform proactive modeling strategies in subjective NLP tasks.

Problem

• Analyzing how language models represent ambiguity in text embeddings
• Exploring topological structures in embedding spaces of fine-tuned models
• Investigating model confidence versus label uncertainty in ambiguous data

Innovation

• Using Mapper tool for text embedding topology
• Analyzing modular non-convex regions in embedding space
• Capturing decision boundaries and overconfident clusters directly