ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentation

2026-08-05

AI Summary

This study addresses the inefficiency and heavy reliance on manual calibration in semantic annotation of artworks, as well as the lack of effective human–AI collaboration mechanisms in existing tools. To overcome these limitations, the authors propose a bidirectional Human–AI Augmentation framework (BiHAA) that implements a closed-loop system, ArtAnno, leveraging a multi-agent architecture to enable real-time interaction and mutual capability enhancement between human experts and AI during annotation. The framework integrates large language models, proactive intelligent support, interaction-driven trajectory distillation, and experience reuse techniques, introducing a novel dynamic mutual reinforcement mechanism. This approach significantly improves annotation efficiency, reduces the verification burden on non-experts, and facilitates continuous accumulation and reuse of domain-specific knowledge.

Abstract

High-quality annotation of artworks is essential for computational art research, yet extracting implicit semantics remains challenging due to the reliance on culturally grounded meanings and deep contextual knowledge behind the images. Current AI-assisted annotation tools often lack assistance or rely on one-way workflows where experts have to perform extra manual calibrations to improve AI models, resulting in limited efficiency. To address this, we propose Bidirectional Human-AI Augmentation(BiHAA), a closed-loop framework in which skills and domain knowledge base evolve through real-time interaction and bidirectional HAI augmentation. Informed by a formative study with 20 artwork annotators from different backgrounds, we implement this framework in ArtAnno, an artwork annotation system driven by a multi-agent architecture. The system includes a Proactive Agentic Support Module, where AI augments humans through semantic mining and label suggestion, and an Interaction-Driven Evolution Module, where human expertise continuously enhances the AI through distilling annotation trajectories into reusable experience. Evaluation through a user study and two case studies demonstrates that our framework and system improve annotation efficiency, enable knowledge accumulation, and reduce the effort of information seeking and verification for annotators with limited domain expertise. We conclude by discussing broader implications and future directions.

Problem

• artwork annotation
• implicit semantics
• human-AI collaboration
• computational art research
• domain knowledge

Innovation

• Bidirectional Human-AI Augmentation
• LLM Agent
• Implicit Semantics Annotation
• Multi-agent Architecture
• Knowledge Evolution