A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generation

2026-09-22

AI Summary

本文提出一种基于符号学框架的方法来评估自然语言生成中的保真度和覆盖率问题，通过符号保真度和符号覆盖率两个指标来衡量文本间的符号对齐情况。

Abstract

When two texts describe the same expression, standard metrics based on lexical overlap or whole-text similarity may fail to detect meaningful differences in how that expression is framed. We propose a framework to evaluate semiotic alignment between texts, where a semiotic profile encompasses both the contextual meaning and the discourse references made salient by a text. Our approach yields two scores, Semiotic Fidelity and Semiotic Coverage, estimating how much of one text's profile is supported by the other and how much of the other's profile it recovers. Experiments show that coverage is typically lower than fidelity, and that alignment between LLMs and human-curated data is highest at low sampling temperatures, while higher temperatures reduce this alignment.

Problem

• semiotic alignment
• natural language generation
• lexical overlap
• text similarity
• fidelity

Innovation

• Semiotic Alignment
• Natural Language Generation
• Fidelity and Coverage
• Sampling Temperature