Reversing Arrows in Large Language Models

2026-08-04

AI Summary

It remains unclear whether large language models (LLMs) can accurately capture directional inverse relations such as “mother–child.” This work presents the first systematic evaluation of mainstream open-source LLMs on this task, introducing a benchmark comprising 27 relation types and 5,457 samples. The evaluation employs a multiple-choice prompting framework augmented with synthetic data generation and masked entity replacement strategies. Findings reveal a systematic asymmetry in models’ comprehension of inverse relations, limited and inconsistent benefits from explicit relation descriptions, and high sensitivity to the surface form of entity representations. This study contributes both a novel evaluation resource and empirical insights into the challenges of modeling relational semantics in LLMs.

Abstract

Large language models (LLMs) have achieved strong performance on text-to-knowledge graph generation and related tasks. Nevertheless, it is still unclear whether they accurately model the direction-dependent semantics of inverse relations, in which reversing the order of the arguments alters the meaning of a relation (e.g., \textit{mother} versus \textit{child}). To the best of our knowledge, this work presents the first systematic study of inverse relation directionality in LLMs, using a benchmark consisting of 5,457 instances spanning 27 distinct inverse relation labels. We evaluate five open-source LLMs under a multiple-choice prompting framework and further examine the influence of relation descriptions and entity representations by substituting the original entities with synthetic and masked entities. Our findings reveal systematic asymmetries in inverse relation classification across LLMs, indicate that relation descriptions do not consistently improve performance, and show that model performance can be sensitive to variations in entity representations.

Problem

• inverse relations
• relation directionality
• large language models
• semantic asymmetry
• knowledge graph

Innovation

• inverse relations
• directionality
• large language models
• relation semantics
• entity representation