🤖 AI Summary
Current evaluations of language models struggle to assess their comprehension of abstract concepts, and the high-dimensional semantic spaces they operate in often lack interpretability. This work introduces topological data analysis into language model evaluation for the first time, proposing a semantic alignment framework that maps low-dimensional, interpretable knowledge structures—such as ontologies and knowledge graphs—onto model embedding spaces. This approach enables cross-lingual and cross-modal tracking of semantic consistency, effectively uncovering the evolutionary dynamics of conceptual representations during model training. Furthermore, it substantially enhances the interpretability of evaluations concerning cross-lingual phrase understanding, offering a principled means to probe how abstract knowledge is encoded and transformed within modern language models.
📝 Abstract
Language model benchmarking is a difficult task. Outcome reasoning alone does not test the model's conceptualization of language and popular open-source benchmarks are quickly saturated or ingested as training data. It is important to test the model's output, but augmenting these tests by characterizing semantic structure gives more insight to how models relate abstract concepts. However, the high dimensional embedding spaces are not easy to interpret. This work demonstrates how topological methods can be used to rigorously compare these spaces to low dimensional and interpretable baselines like ontologies and curated knowledge graphs. These multi-modal alignment tests make it possible to track model adaptations and test phrase understanding across multiple languages.