CKnowEdit: A New Chinese Knowledge Editing Dataset for Linguistics, Facts, and Logic Error Correction in LLMs

2024-09-09

AI Summary

Mainstream large language models (LLMs) exhibit significant deficiencies in understanding and correcting Chinese cultural knowledge—such as classical poetry, idioms (chengyu), and proverbs—due to the lack of culturally grounded evaluation benchmarks. Method: We introduce CEval-Chinese, the first knowledge editing evaluation dataset tailored to Chinese linguistic and cultural characteristics. It covers seven categories of cultural knowledge and systematically models Chinese-specific challenges—including antithetical parallelism, polyphonic characters, and nested logical structures—using multi-source data from classical texts and online communities, rigorously validated by human annotators and domain experts. The dataset comprises high-quality samples with three distinct error types. Contribution/Results: Experiments reveal systematic semantic-level failures of current LLMs on cultural knowledge and expose critical performance bottlenecks of mainstream knowledge editing methods in Chinese contexts. CEval-Chinese fills a key gap in Chinese knowledge editing benchmarks and establishes a new paradigm for evaluating and enhancing culture-aware language capabilities.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Chinese, as a linguistic system rich in depth and complexity, is characterized by distinctive elements such as ancient poetry, proverbs, idioms, and other cultural constructs. However, current Large Language Models (LLMs) face limitations in these specialized domains, highlighting the need for the development of comprehensive datasets that can assess, continuously update, and progressively improve these culturally-grounded linguistic competencies through targeted training optimizations. To address this gap, we introduce CKnowEdit, the first-ever Chinese knowledge editing dataset designed to correct linguistic, factual, and logical errors in LLMs. We collect seven types of knowledge from a wide range of sources, including classical texts, idioms, and content from Baidu Tieba Ruozhiba, taking into account the unique polyphony, antithesis, and logical structures inherent in the Chinese language. By analyzing this dataset, we highlight the challenges current LLMs face in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge editing techniques reveals opportunities to advance the correction of Chinese knowledge. Code and dataset are available at https://github.com/zjunlp/EasyEdit.

Problem

• Chinese knowledge editing
• linguistic, factual, logical errors
• Large Language Models (LLMs)

Innovation

• Chinese knowledge editing dataset
• corrects linguistic, factual errors
• advances state-of-the-art techniques