PoemTale Diffusion: Minimising Information Loss in Poem to Image Generation with Multi-Stage Prompt Refinement

2025-07-18

AI Summary

Current text-to-image diffusion models often suffer from semantic information loss when generating images from poetry, due to their inability to adequately parse abstract, polysemous poetic language. To address this, we propose a training-free, multi-stage prompt refinement framework that leverages large language models for iterative semantic parsing and enhancement, and modifies the diffusion model’s self-attention mechanism by introducing consistency-aware self-attention to ensure semantic coherence across generated image sequences. This approach significantly improves modeling of deep poetic features—such as metaphor and layered imagery. Experiments on our curated P4I dataset (1,111 poems) demonstrate substantial improvements over baselines: +23.6% in semantic alignment (measured by CLIP-Score) and +37.2% in artistic expressiveness (as rated by domain experts). The method establishes a new, interpretable, and high-fidelity paradigm for poetic visualization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent advancements in text-to-image diffusion models have achieved remarkable success in generating realistic and diverse visual content. A critical factor in this process is the model's ability to accurately interpret textual prompts. However, these models often struggle with creative expressions, particularly those involving complex, abstract, or highly descriptive language. In this work, we introduce a novel training-free approach tailored to improve image generation for a unique form of creative language: poetic verse, which frequently features layered, abstract, and dual meanings. Our proposed PoemTale Diffusion approach aims to minimise the information that is lost during poetic text-to-image conversion by integrating a multi stage prompt refinement loop into Language Models to enhance the interpretability of poetic texts. To support this, we adapt existing state-of-the-art diffusion models by modifying their self-attention mechanisms with a consistent self-attention technique to generate multiple consistent images, which are then collectively used to convey the poem's meaning. Moreover, to encourage research in the field of poetry, we introduce the P4I (PoemForImage) dataset, consisting of 1111 poems sourced from multiple online and offline resources. We engaged a panel of poetry experts for qualitative assessments. The results from both human and quantitative evaluations validate the efficacy of our method and contribute a novel perspective to poem-to-image generation with enhanced information capture in the generated images.

Problem

• Minimizing information loss in poem-to-image generation
• Improving interpretability of poetic texts for image models
• Generating consistent images to convey layered poem meanings

Innovation

• Multi-stage prompt refinement for poetic texts
• Consistent self-attention for multiple image generation
• P4I dataset introduction for poetry research