CuratorMAS: Automating Dataset Curation via Multi-Agent Orchestration

2026-10-05

AI Summary

This study addresses the high costs, limited cross-domain generalization, and reliance on manual rules in dataset curation by proposing a multi-agent collaborative framework. The framework decouples the curation pipeline into five programmable stages and introduces a pioneering parallelized workflow that integrates online knowledge retrieval with skill evolution modules. Through multi-agent orchestration, context exploration, dynamic metric computation, and self-evolution mechanisms, it enables automated evaluation and filtering. Experimental results demonstrate that the proposed method reduces data noise rates by 36.03 percentage points and improves downstream model F1 scores by 8.88 percentage points, significantly enhancing the flexibility and adaptability of the curation process.

Abstract

High-quality datasets are essential for reliable machine learning, but dataset curation remains costly and hard to generalize across domains. Existing methods typically rely on manually designed heuristics or model-dependent signals, limiting their applicability across tasks and user queries. To address these limitations and automate data curation, we propose \textbf{CuratorMAS}, a multi-agent collaboration framework that orchestrates agents to evaluate and curate high-quality datasets. To achieve the goal of flexible curation, CuratorMAS decomposes the complex curation process into five programmable execution stages and forms a parallelizable workflow. Specifically, CuratorMAS first performs dataset exploration to collect contextual information such as file structures and constraint cues, thereby developing a comprehensive understanding of the given task. In order to acquire up-to-date information, CuratorMAS retrieves domain knowledge from online sources to augment the evaluation process. Next, CuratorMAS derives the necessary evaluation criteria and computes the corresponding metrics. Based on these results, CuratorMAS executes filtering accordingly. Finally, an evolution module summarizes the evaluation outcomes and updates the relevant skills. Extensive and comprehensive experiments demonstrate that CuratorMAS significantly reduces the noise rate by up to 36.03 percentage points (pp) while also improving the F1 score of downstream models by up to 8.88 pp.

Problem

• Dataset Curation
• Data Quality
• Multi-Agent Systems
• Automation
• Domain Generalization

Innovation

• Multi-Agent Orchestration
• Dataset Curation
• Automated Evaluation
• Knowledge Retrieval
• Skill Evolution