Automated Feedback Loops to Protect Text Simplification with Generative AI from Information Loss

2025-05-22

AI Summary

Generative AI often omits critical information when simplifying health texts, compromising comprehensibility. To address this, we propose the first automated feedback-loop framework for health information simplification, comprising three tightly integrated stages: omission detection, importance assessment, and targeted repair. Our key finding reveals that injecting *all* detected missing entities—rather than applying ranked filtering or random filling—yields significantly superior simplification quality, exposing fundamental limitations in current LLMs’ ability to rank entity importance accurately. We implement simplification and omission analysis using GPT-4-0613, and evaluate semantic fidelity at both summary- and full-text granularity via cosine similarity and ROUGE-L. Evaluated on 50 real-world health text pairs, our method achieves substantial improvements over baselines: +12.7% in semantic similarity and +18.3% in content overlap.

Technology Category

• Natural Language Processing: Summarization
• Humans and AI: AI for Accessibility
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Understanding health information is essential in achieving and maintaining a healthy life. We focus on simplifying health information for better understanding. With the availability of generative AI, the simplification process has become efficient and of reasonable quality, however, the algorithms remove information that may be crucial for comprehension. In this study, we compare generative AI to detect missing information in simplified text, evaluate its importance, and fix the text with the missing information. We collected 50 health information texts and simplified them using gpt-4-0613. We compare five approaches to identify missing elements and regenerate the text by inserting the missing elements. These five approaches involve adding missing entities and missing words in various ways: 1) adding all the missing entities, 2) adding all missing words, 3) adding the top-3 entities ranked by gpt-4-0613, and 4, 5) serving as controls for comparison, adding randomly chosen entities. We use cosine similarity and ROUGE scores to evaluate the semantic similarity and content overlap between the original, simplified, and reconstructed simplified text. We do this for both summaries and full text. Overall, we find that adding missing entities improves the text. Adding all the missing entities resulted in better text regeneration, which was better than adding the top-ranked entities or words, or random words. Current tools can identify these entities, but are not valuable in ranking them.

Problem

• Detecting missing information in AI-simplified health texts
• Evaluating importance of omitted content in simplified text
• Regenerating simplified text by reinserting crucial missing elements

Innovation

• Automated feedback loops detect missing information
• Generative AI evaluates and fixes simplified text
• Adding missing entities improves text regeneration