Evaluating Chinese Ambiguity Understanding in Large Language Models

2026-05-15

AI Summary

This study addresses the underexplored challenge of Chinese ambiguity comprehension in large language models (LLMs) and the absence of high-quality, scalable Chinese ambiguity datasets. It presents the first application of Potential Ambiguity (PA) theory to Chinese, introducing a semi-automatic corpus generation method to construct CHA-Gen—a dataset comprising 5,712 annotated sentences—and systematically evaluates the ambiguity detection capabilities of prevailing LLMs. Experimental results reveal that current models generally struggle to accurately identify Chinese ambiguities; however, Chain-of-Thought (CoT) prompting consistently enhances performance. Notably, base models outperform instruction-tuned counterparts in capturing semantic diversity and exhibit a stronger tendency to favor dominant interpretations. This work contributes novel data, methodology, and empirical insights to advance research on Chinese ambiguity understanding.

Abstract

Linguistic ambiguity is critical to the robustness of Large Language Models (LLMs), yet existing research focuses mostly on English, with limited attention devoted to Chinese. Existing Chinese ambiguity datasets (e.g., CHAmbi) suffer from poor scalability. Guided by Potential Ambiguity (PA) Theory, we design a semi-automatic pipeline to construct CHA-Gen. It is the first PA Theory-grounded Chinese ambiguity dataset, which comprises 5,712 sentences (2,414 ambiguous, 3,298 unambiguous) across 18 potential ambiguous structures. Evaluating LLMs (e.g. Gemma 3, Qwen 2.5/3 series) via direct querying and machine translation, we find that LLMs struggle with ambiguity detection (improved by CoT prompting). Analysis of Qwen3-32B's CoT rationales reveals three common failure modes: ambiguity blindness, misattribution, and premature resolution. Uncertainty quantification with semantic entropy metric shows higher uncertainty for ambiguous sentences. Moreover, instruction tuning induces overconfidence, whereas Base models better capture semantic diversity. We further observe that models exhibit a bias toward dominant interpretations. Our work provides a scalable approach for Chinese ambiguity corpus and insights into LLMs' ambiguity handling, laying a foundation for enhancing Chinese ambiguity research in LLMs.

Problem

• Chinese ambiguity
• Large Language Models
• linguistic ambiguity
• scalability
• ambiguity understanding

Innovation

• Potential Ambiguity Theory
• semi-automatic dataset construction
• Chinese linguistic ambiguity
• semantic entropy
• chain-of-thought prompting