HEPHA: A Mixed-Initiative Image Labeling Tool for Specialized Domains

2025-03-05

AI Summary

To address the high annotation cost and scarcity of domain experts in specialized fields such as healthcare, this paper proposes VISA, a hybrid active annotation framework that integrates visual predicate modeling, inductive logic programming (ILP), and active learning. VISA enables zero-programming users to iteratively generate and refine interpretable logical rules via intuitive image-based interactions. Its key innovation lies in a dual-layer recommendation mechanism—combining a visual programming interface with active sampling—to jointly recommend both predicates and rule candidates, thereby significantly reducing expert intervention frequency. A user study with 16 participants demonstrates that VISA improves annotation accuracy by 27% on domain-specific tasks and by 19% on general-domain tasks, while delivering highly interpretable rules and achieving a 3.2× increase in annotation efficiency.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Active Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Image labeling is an important task for training computer vision models. In specialized domains, such as healthcare, it is expensive and challenging to recruit specialists for image labeling. We propose HEPHA, a mixed-initiative image labeling tool that elicits human expertise via inductive logic learning to infer and refine labeling rules. Each rule comprises visual predicates that describe the image. HEPHA enables users to iteratively refine the rules by either direct manipulation through a visual programming interface or by labeling more images. To facilitate rule refinement, HEPHA recommends which rule to edit and which predicate to update. For users unfamiliar with visual programming, HEPHA suggests diverse and informative images to users for further labeling. We conducted a within-subjects user study with 16 participants and compared HEPHA with a variant of HEPHA and a deep learning-based approach. We found that HEPHA outperforms the two baselines in both specialized-domain and general-domain image labeling tasks. Our code is available at https://github.com/Neural-Symbolic-Image-Labeling/NSILWeb.

Problem

• Reduces cost of recruiting specialists for image labeling in specialized domains.
• Improves image labeling accuracy using inductive logic learning and rule refinement.
• Facilitates user-friendly rule updates via visual programming and image recommendations.

Innovation

• Mixed-initiative tool for image labeling
• Inductive logic learning for rule inference
• Visual programming interface for rule refinement