Reasoning or Memorization: Can LLMs Understand and Generate Chinese Xiehouyu Riddles?

2026-07-25

AI Summary

This study investigates whether large language models rely more on memorization or reasoning when comprehending and generating Chinese xiehouyu (a form of witty, two-part allegorical sayings), while also evaluating their creative capacity. To mitigate training data contamination, the authors construct a novel dataset comprising low-frequency and entirely new xiehouyu crafted by linguists, and design multiple-choice comprehension, free-form explanation, and generation tasks. They introduce a new metric, Δacc—the accuracy difference between known and novel items—to quantify model reliance on memorization. Experimental results reveal that state-of-the-art Chinese-language models exhibit a Δacc of 23.6%, substantially higher than that of English models (5.1%), indicating greater dependence on memorization. Notably, Gemini 3.1 Pro achieves 92.6% accuracy on novel xiehouyu comprehension, outperforming humans by 24%; however, its generated xiehouyu remain significantly less fluent and creative than human-authored ones.

Abstract

In this paper, we push the boundary of LLM reasoning by testing them in a Chinese language game, xiehouyu, with novel xiehouyu created by linguists that had not existed before to avoid data contamination. We use multiple-choice questions (MCQ), free-form explanation generation, and new xiehouyu creation to evaluate LLMs' ability to understand and create xiehouyu. In MCQ, we use the delta of accuracy ($Δ_{acc}$) between existing but low-frequency xiehouyu and novel ones as an index for memorization. $Δ_{acc}$ for native speakers is very low, suggesting similar processing mechanisms. However, we found that frontier Chinese models have on average a $Δ_{acc}$ of 23.6\%, while English-centric models tested have a mean $Δ_{acc}$ of 5.1\%, suggesting that frontier Chinese models are likely trained with much larger Chinese data, thus memorizing more low-frequency xiehouyu. For novel xiehouyu, Gemini 3.1 Pro demonstrated remarkable ability with acc 92.6, which is 24\% higher than human accuracy. In xiehouyu creation, those created by LLMs receive much worse ratings than those by humans. These results suggest that claims about the reasoning abilities of LLMs may need careful re-examination considering the data contamination issue, and that LLMs' creativity in language-related tasks may still be behind human experts, at least in Chinese xiehouyu.

Problem

• reasoning
• memorization
• xiehouyu
• data contamination
• language models

Innovation

• xiehouyu
• reasoning vs memorization
• data contamination
• Δ_acc metric
• LLM creativity