Dataforge: A Data Agent Platform for Autonomous Data Engineering

2025-11-09

AI Summary

Data preparation for AI applications in materials discovery, molecular modeling, and climate science remains labor-intensive, expert-dependent, and poorly scalable. Method: This paper introduces the first end-to-end autonomous data agent system, featuring dual feedback loops that integrate large language models’ (LLMs) semantic understanding with domain-knowledge-guided grounded validation. It employs hierarchical routing, automated cleaning and normalization, and feature-level optimization algorithms to transform raw tabular data into AI-ready datasets without human intervention. Contribution/Results: Evaluated on real-world datasets across multiple scientific domains, the system significantly improves data quality and downstream model performance. It establishes the first safe, reliable, and non-expert-friendly paradigm for autonomous data engineering—enabling fully automated, robust, and interpretable data curation tailored to domain-specific requirements.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Application Domains: Humanities & Computational Social Science

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data integration and cleaning

Abstract

The growing demand for AI applications in fields such as materials discovery, molecular modeling, and climate science has made data preparation an important but labor-intensive step. Raw data from diverse sources must be cleaned, normalized, and transformed to become AI-ready, while effective feature transformation and selection are essential for efficient training and inference. To address the challenges of scalability and expertise dependence, we present Data Agent, a fully autonomous system specialized for tabular data. Leveraging large language model (LLM) reasoning and grounded validation, Data Agent automatically performs data cleaning, hierarchical routing, and feature-level optimization through dual feedback loops. It embodies three core principles: automatic, safe, and non-expert friendly, which ensure end-to-end reliability without human supervision. This demo showcases the first practical realization of an autonomous Data Agent, illustrating how raw data can be transformed"From Data to Better Data."

Problem

• Automating data cleaning and transformation for AI applications
• Reducing labor-intensive data preparation through autonomous systems
• Enabling non-experts to perform reliable feature engineering

Innovation

• Autonomous system for tabular data processing
• LLM reasoning with grounded validation techniques
• Automatic data cleaning and feature optimization