Emerging Reliance Behaviors in Human-AI Text Generation: Hallucinations, Data Quality Assessment, and Cognitive Forcing Functions

2024-09-13

arXiv.org

AI Summary

This study investigates how large language model (LLM) hallucination and cognitive forcing jointly influence user dependency behavior and data quality during human-LLM collaborative generation of customer service dialogues. Through a user behavior experiment (N=11, 88 tasks), LLM dialogue generation, qualitative coding, and quantitative analysis, we empirically demonstrate— for the first time—a significant interaction effect: hallucination substantially degrades data quality; while cognitive forcing does not universally mitigate hallucination’s adverse effects, it systematically reshapes user adoption strategies, giving rise to three distinct dependency patterns. Our core contributions are threefold: (1) uncovering the synergistic mechanism between hallucination and cognitive forcing; (2) proposing a novel data quality assessment paradigm anchored in observable user behavior; and (3) providing empirical foundations for designing robust, trustworthy human-AI collaborative data production pipelines resilient to hallucination-induced interference.

Technology Category

• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

In this paper, we investigate the impact of hallucinations and cognitive forcing functions in human-AI collaborative text generation tasks, focusing on the use of Large Language Models (LLMs) to assist in generating high-quality conversational data. LLMs require data for fine-tuning, a crucial step in enhancing their performance. In the context of conversational customer support, the data takes the form of a conversation between a human customer and an agent and can be generated with an AI assistant. In our inquiry, involving 11 users who each completed 8 tasks, resulting in a total of 88 tasks, we found that the presence of hallucinations negatively impacts the quality of data. We also find that, although the cognitive forcing function does not always mitigate the detrimental effects of hallucinations on data quality, the presence of cognitive forcing functions and hallucinations together impacts data quality and influences how users leverage the AI responses presented to them. Our analysis of user behavior reveals distinct patterns of reliance on AI-generated responses, highlighting the importance of managing hallucinations in AI-generated content within conversational AI contexts.

Problem

• Impact of hallucinations on human-AI data generation quality
• Role of Cognitive Forcing Functions in user-AI response integration
• Overreliance on AI-generated text reduces collaborative output quality

Innovation

• Using LLMs for conversational data generation
• Studying Cognitive Forcing Functions' effects
• Analyzing overreliance on AI-generated content