Human-In-The-Loop Workflow for Neuro- Symbolic Scholarly Knowledge Organization

2025-06-03

AI Summary

To address the low efficiency of knowledge structuring in scientific literature—exacerbated by its exponential growth and heavy reliance on expert curation—this paper proposes a neural-symbolic, human-in-the-loop (HITL) workflow. The method leverages large language models (LLMs) to automatically extract and structure scholarly information, which is then ingested into the Open Research Knowledge Graph (ORKG). A modular architecture enables customizable LLM selection and multi-stage human verification, tightly coupling automation with expert oversight. Its key innovation lies in pioneering a synergistic mechanism between LLMs and symbolic knowledge graphs within the HITL paradigm. Evaluation shows the system achieves a System Usability Scale (SUS) score of 84.17 (A+ level), and reduces per-paper knowledge modeling time from hours to weeks down to an average of 24 minutes and 40 seconds—demonstrating substantial gains in scientific knowledge transformation efficiency.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

As the volume of scientific literature continues to grow, efficient knowledge organization is an increasingly challenging task. Traditional structuring of scientific content is time-consuming and requires significant domain expertise, increasing the need for tool support. Our goal is to create a Human-in-the-Loop (HITL) workflow that supports researchers in creating and structuring scientific knowledge, leveraging neural models and knowledge graphs, exemplified using the Open Research Knowledge Graph (ORKG). The workflow aims to automate key steps, including data extraction and knowledge structuring, while keeping user oversight through human validation. We developed a modular framework implementing the workflow and evaluated it along the Quality Improvement Paradigm (QIP) with participants from the ORKG user community. The evaluation indicated that the framework is highly usable and provides practical support. It significantly reduces the time and effort required to transition from a research interest to literature-based answers by streamlining the import of information into a knowledge graph. Participants evaluated the framework with an average System Usability Scale (SUS) score of 84.17, an A+ -- the highest achievable rating. They also reported that it improved their time spent, previously between 4 hours and two weeks, down to an average of 24:40 minutes. The tool streamlines the creation of scientific corpora and extraction of structured knowledge for KG integration by leveraging LLMs and user-defined models, significantly accelerating the review process. However, human validation remains essential throughout the extraction process, and future work is needed to improve extraction accuracy and entity linking to existing knowledge resources.

Problem

• Automate scientific knowledge structuring using neural models and knowledge graphs
• Reduce time and effort for literature-based research answers
• Streamline creation of scientific corpora and KG integration

Innovation

• Human-in-the-Loop workflow with neural models
• Modular framework for knowledge graph integration
• LLMs and user-defined models for extraction