KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms

2026-09-17

AI Summary

本文针对韩语新词理解问题，构建了包含1785个新词的KoNeoBench数据集，并通过四个任务评估了大语言模型在处理这些新词时的表现。

Abstract

Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging words and meanings. Existing Korean benchmarks are centered on established vocabulary and therefore provide limited coverage of such recent lexical change, and their English-oriented design makes it difficult to assess the typological properties of Korean, in which content words combine productively with functional morphemes. In this paper, we introduce KoNeoBench, a benchmark for evaluating LLMs' understanding of Korean neologisms. KoNeoBench is built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review. Each entry provides usage examples, word-formation analyses, and dictionary-style definitions. Based on this resource, we define four tasks and report results on recent models, together with a human baseline. Our experiments show that current LLMs exhibit clear limitations in recovering source components, distinguishing semantic categories, and generating accurate definitions. These results reveal specific aspects of recent Korean lexical change that remain challenging for current LLMs. KoNeoBench is available at https://github.com/bcmilab/ko-neobench/ .

Problem

• Korean neologisms
• large language models
• lexical change
• typological properties

Innovation

• Korean neologisms
• LLM evaluation
• lexical change