🤖 AI Summary
Current large language models (LLMs) exhibit significant limitations in interpreting non-literal semantics, particularly idioms and metaphors. To address this, we introduce IdiomMetaphorBank—the first large-scale, multi-category, human-annotated dataset for idiom and metaphor understanding. Our methodology employs a scalable data framework integrating corpus-driven automatic extraction with expert-level manual annotation, augmented by context-aware, model-agnostic post-processing to support both slot-filling and sequence labeling tasks. Empirically, IdiomMetaphorBank improves F1 scores of mainstream pretrained models on idiom identification by 12.3%. Moreover, it enables, for the first time, fine-grained evaluation of implicit semantic detection—e.g., underlying conceptual mappings and figurative intent—thereby establishing a benchmark resource and methodological foundation for non-literal semantic modeling.
📝 Abstract
Idiomatic and figurative language form a large portion of colloquial speech and writing. With social media, this informal language has become more easily observable to people and trainers of large language models (LLMs) alike. While the advantage of large corpora seems like the solution to all machine learning and Natural Language Processing (NLP) problems, idioms and figurative language continue to elude LLMs. Finetuning approaches are proving to be optimal, but better and larger datasets can help narrow this gap even further. The datasets presented in this paper provide one answer, while offering a diverse set of categories on which to build new models and develop new approaches. A selection of recent idiom and figurative language datasets were used to acquire a combined idiom list, which was used to retrieve context sequences from a large corpus. One large-scale dataset of potential idiomatic and figurative language expressions and two additional human-annotated datasets of definite idiomatic and figurative language expressions were created to evaluate the baseline ability of pre-trained language models in handling figurative meaning through idiom recognition (detection) tasks. The resulting datasets were post-processed for model agnostic training compatibility, utilized in training, and evaluated on slot labeling and sequence tagging.