On Improving Faithfulness of Podcasts from Documents

2026-07-24

AI Summary

This work addresses the issue of hallucination in large language models (LLMs) when generating document-driven, multi-speaker podcast dialogues, where models often introduce ungrounded information that compromises content fidelity. The study presents the first systematic investigation into faithfulness in long-form, multi-turn podcast generation, introducing a dataset of over 1,500 source documents spanning five domains and a turn-level LLM-as-a-judge evaluation framework. To mitigate unfaithful generations without sacrificing conversational fluency, the authors propose Catch-n-Repair, a model-agnostic mechanism that detects and corrects hallucinated content. Experimental results demonstrate that this approach significantly improves generation faithfulness in both in-domain and cross-domain settings, confirming its effectiveness and generalizability.

Abstract

Large language models (LLMs) are increasingly used to generate long-form conversational content such as podcasts from textual sources. While these systems produce fluent and engaging narratives, they often introduce ungrounded information. In this work, we present the first systematic study of faithfulness in document-grounded podcast generation, where grounding must be maintained across conversational turns in long-form, multi-speaker transcripts. We construct a dataset of over 1500 documents spanning five domains and generate podcast transcripts using multiple LLMs. We introduce a turn-level LLM-as-a-judge framework for evaluating whether conversational turns are supported by the source document, and validate its reliability through human studies. Our analysis shows that even state-of-the-art models, including GPT-4o, frequently generate ungrounded content. To mitigate this issue, we propose catch-n-repair, a model-agnostic framework that detects and rewrites unfaithful conversational turns while preserving conversational flow. Experiments demonstrate consistent improvements in faithfulness across both in-domain and out-of-domain settings.

Problem

• faithfulness
• document-grounded generation
• podcast generation
• hallucination
• conversational AI

Innovation

• faithfulness
• document-grounded generation
• LLM-as-a-judge
• catch-n-repair
• conversational podcast generation