DatasetAgent: A Novel Multi-Agent System for Auto-Constructing Datasets from Real-World Images

2025-07-11

AI Summary

To address the labor-intensive, inefficient, and inflexible nature of constructing real-world image datasets, this paper introduces the first multi-agent collaborative framework powered by multimodal large language models (MLLMs). The framework integrates four specialized agents—retrieval, filtering, annotation, and augmentation—coupled with an image optimization toolkit, enabling both extension of existing datasets and zero-shot generation of novel ones. Fully automated and user-controllable, it supports customizable data distributions and task specifications. Extensive evaluation across multiple open-source benchmarks demonstrates that datasets generated by our framework substantially improve model performance in image classification, object detection, and semantic segmentation, yielding average gains of 3.2–5.7 percentage points in Top-1 accuracy or mAP. This work establishes a scalable, reproducible paradigm for autonomous construction of high-quality vision datasets.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Common knowledge indicates that the process of constructing image datasets usually depends on the time-intensive and inefficient method of manual collection and annotation. Large models offer a solution via data generation. Nonetheless, real-world data are obviously more valuable comparing to artificially intelligence generated data, particularly in constructing image datasets. For this reason, we propose a novel method for auto-constructing datasets from real-world images by a multiagent collaborative system, named as DatasetAgent. By coordinating four different agents equipped with Multi-modal Large Language Models (MLLMs), as well as a tool package for image optimization, DatasetAgent is able to construct high-quality image datasets according to user-specified requirements. In particular, two types of experiments are conducted, including expanding existing datasets and creating new ones from scratch, on a variety of open-source datasets. In both cases, multiple image datasets constructed by DatasetAgent are used to train various vision models for image classification, object detection, and image segmentation.

Problem

• Automating dataset construction from real-world images
• Reducing manual effort in image collection and annotation
• Enhancing dataset quality using multi-agent collaboration

Innovation

• Multi-agent system for auto-constructing real-world image datasets
• Utilizes Multi-modal Large Language Models (MLLMs)
• Integrates image optimization tools for dataset enhancement