Technology Category
Natural Language Processing: Question AnsweringData Mining & Knowledge Management: Conversational Systems for Recommendation & RetrievalCognitive Modeling & Cognitive Systems: Simulating Human Behavior
DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering
🤖 AI Summary
This study addresses the lack of systematic representation and comparison of rhetorical strategies employed by humans and large language models (LLMs) in retrieval-based question answering. It proposes DiscoTrace, a novel method grounded in Rhetorical Structure Theory that models answers as sequences of rhetorical acts paired with question interpretations, thereby establishing the first structured framework enabling direct comparison of response strategies across human communities and LLMs. Integrating discourse act annotation, question interpretation modeling, and cross-group analysis, the approach reveals significant variation in rhetorical strategies among nine distinct human communities. In contrast, even under imitation prompting, LLMs exhibit limited rhetorical diversity and consistently favor broad coverage over selective engagement with specific question interpretations.
Technology Category
Application Category
📝 Abstract
We introduce DiscoTrace, a method to identify the rhetorical strategies that answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different human communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. Our findings can guide the development of pragmatic LLM answerers that consider a range of strategies informed by context in QA.
Problem
Research questions and friction points this paper is trying to address.
rhetorical strategies
information-seeking question answering
discourse acts
LLM answer diversity
human vs. LLM
Innovation
Methods, ideas, or system contributions that make the work stand out.
DiscoTrace
rhetorical strategy
discourse act
information-seeking QA
LLM pragmatics
🔎 Similar Papers
No similar papers found.
