DALL: Data Labeling via Data Programming and Active Learning Enhanced by Large Language Models

2026-02-15

AI Summary

This work proposes an interactive text annotation system that integrates data programming, active learning, and large language models (LLMs) to address the high cost and low efficiency of acquiring high-quality labeled data in natural language processing. The system introduces a novel, code-free approach for defining structured labeling functions and leverages LLMs to dynamically refine labels and iteratively improve these functions, thereby enabling efficient human-in-the-loop annotation. Experimental results demonstrate that the proposed method significantly outperforms baseline approaches in terms of annotation efficiency, module effectiveness, and user usability, leading to substantial improvements in both labeling quality and development productivity.

Technology Category

Natural Language Processing: Code Generation / Program Synthesis from Natural LanguageMachine Learning: Active LearningHumans and AI: Human-in-the-loop Machine Learning

Application Category

Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labelingSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Deep learning models for natural language processing rely heavily on high-quality labeled datasets. However, existing labeling approaches often struggle to balance label quality with labeling cost. To address this challenge, we propose DALL, a text labeling framework that integrates data programming, active learning, and large language models. DALL introduces a structured specification that allows users and large language models to define labeling functions via configuration, rather than code. Active learning identifies informative instances for review, and the large language model analyzes these instances to help users correct labels and to refine or suggest labeling functions. We implement DALL as an interactive labeling system for text labeling tasks. Comparative, ablation, and usability studies demonstrate DALL's efficiency, the effectiveness of its modules, and its usability.

Problem

• data labeling
• label quality
• labeling cost
• natural language processing
• deep learning

Innovation

• data programming
• active learning
• large language models
• interactive labeling
• labeling functions