Scaling Expert Feedback with Reflective Edit Propagation in Compositional Knowledge Bases

2026-06-03

AI Summary

This work addresses the inefficiency and limited scalability of expert-driven validation in large-scale domain knowledge bases, where human experts must manually verify content generated by large language models (LLMs). To overcome this bottleneck, the authors propose RAID, a novel system that departs from the conventional “edit-and-save” paradigm. RAID employs a three-stage architecture—intent inference, reflective planning, and user-controllable execution—to distill a single expert edit into a generalizable semantic intent and propagate this correction across the entire knowledge base. This approach enables the first systematic reuse and scaling of expert knowledge. Experimental results demonstrate that RAID accurately captures expert intent on both public and proprietary datasets, significantly improving the efficiency of knowledge base maintenance.

Abstract

Domain-specific knowledge bases (KBs) encode vertical expertise and proprietary information that organizations depend on, but curating them at scale is a persistent challenge. Although Large Language Models (LLMs) can draft initial entries efficiently, technical accuracy still requires human expert validation, and reviewing entries one by one at scale is impractical. We present Reflective Agent for Identifier Dictionary (RAID), a novel system that transforms individual expert edits into systematic knowledge updates. Unlike traditional "correct-and-save" paradigms, RAID utilizes a reflective agent to infer the underlying semantic intent behind a single expert edit and propagates that correction across the entire KB through a three-step architecture: Intent Inference, Reflection-based Planning, and User Controlled Execution. We evaluated the reflection and propagation performance on a public dataset and conducted a user study with subject matter experts with proprietary data. The evaluation shows RAID's technical feasibility in capturing expert intent and its potential to scale specialized expertise across industrial knowledge bases.

Problem

• knowledge bases
• expert feedback
• scaling curation
• technical accuracy
• large language models

Innovation

• Reflective Agent
• Edit Propagation
• Intent Inference
• Compositional Knowledge Base
• Expert Feedback Scaling