🤖 AI Summary
Low data quality in unstructured text (e.g., clinical notes) stems from semantic incompleteness, hindering reliable downstream analysis. Method: This paper proposes a data cleaning framework based on *updatable extraction views*, integrating rule-driven information extraction with formal modeling of document update semantics to construct verifiable and interpretable cleaning rules. It establishes, for the first time, sufficient conditions linking extraction program upgradability to cleaning effectiveness. Contribution/Results: Evaluated on real-world medical records, the framework significantly improves entity consistency and completeness, achieving a cleaning accuracy of 92.7%. It ensures verifiability and full traceability of the cleaning process—critical for high-stakes domains—and introduces a novel paradigm for unstructured data quality assurance that bridges theoretical rigor with engineering practicality.
📝 Abstract
Improving data quality in unstructured documents is a long-standing challenge. Unstructured data, especially in textual form, inherently lacks defined semantics, which poses significant challenges for effective processing and for ensuring data quality. We propose leveraging information extraction algorithms to design, apply, and explain data cleaning processes for documents. Specifically, for a simple document update model, we identify and verify a set of sufficient conditions for rule-based extraction programs to qualify for inclusion in our document cleaning framework. Through experiments conducted on medical records, we demonstrate that our approach provides an effective framework for identifying and correcting data quality problems, thereby highlighting its practical value in real-world applications.