pandas dataframe manipulation

Traditional data preparation methods face limitations in semantic understanding and generalization, struggling to meet the rapidly growing demand for application-ready data. This work systematically reviews the application of large language models (LLMs) in three core tasks—data cleaning, integration, and augmentation—and proposes a task-centered taxonomy that, for the first time, delineates the evolutionary trajectory of LLM-driven data preparation techniques. Through a comprehensive literature review, the study examines key technologies such as prompt engineering, agent-based architectures, and semantic matching, alongside prevailing datasets and evaluation metrics. It highlights LLMs’ strengths in enhancing generalization and semantic comprehension while identifying critical challenges related to computational cost, hallucination, scalability, and the lack of standardized evaluation frameworks. The paper concludes by outlining a roadmap for future research and development in this emerging field.

This work addresses the significant limitations of spreadsheet-based analysis in reproducibility, auditability, version control, and automation. It proposes a migration pathway from Excel to research-grade analytical workflows by leveraging Python’s pandas library as a bridge. The study introduces an innovative set of Excel-to-pandas mapping rules, categorizes nine canonical workflow patterns, and compiles a catalog of common failure modes. Seven end-to-end real-world examples demonstrate the approach in practice. By retaining Excel as a familiar interface for input and output while integrating version control, automated refreshing, and seamless incorporation of statistical and machine learning methods, the proposed framework enables governed, reproducible, and auditable tabular data analysis.

Data standardization is critical in the data science lifecycle, yet existing tools (e.g., Pandas) require manual, error-prone coding, while LLM-based automation still demands expert prompt engineering and iterative interaction. To address this, we propose a declarative API-driven LLM-Agent framework that introduces *Dataprep.Clean*—a novel, column-type-aware standardization component enabling end-to-end automation via a single-line operation and one-shot natural language input. Our method integrates domain knowledge modeling with lightweight agent orchestration, eliminating programming prerequisites and enabling semantic cleaning of heterogeneous columns. Evaluated on real-world datasets, the approach achieves high accuracy and robustness across diverse standardization tasks. Deployed as an interactive web tool, it substantially lowers the barrier to entry for data practitioners. This work advances data preprocessing toward declarative, intelligent automation—bridging the gap between domain expertise and scalable, user-friendly tooling.

To address the lack of early, machine-readable descriptions of scientific data analysis workflows—hindering FAIR (Findable, Accessible, Interoperable, Reusable) compliance—this paper introduces dtreg, the first structured registration framework for statistical and machine learning pipelines targeting the pre-publication stage and supporting both Python and R. Its core contributions are: (1) a novel pre-analysis metadata registration mechanism; (2) a persistent, globally identifiable schema system covering mainstream statistical tests (e.g., t-tests) and ML methods; and (3) lightweight, automated RDF/Linked Data serialization to Turtle and JSON-LD. Leveraging object-oriented modeling, dynamic schema population, and export capabilities, dtreg enables end-to-end machine-readable workflow documentation. As an open-source infrastructure, it significantly enhances the findability, interoperability, and reusability of analytical methods in computational research.

This work addresses the problem of labor-intensive, non-generalizable tabular data cleaning. We propose the first end-to-end autonomous agent framework powered by large language models (LLMs), which cleans data solely by generating and executing Python code—without altering downstream ML training pipelines or feature engineering. The agent integrates intra-row contextual understanding with iterative feedback reinforcement, enabling automatic detection and correction of single-row logical errors and outliers, without predefined rules or domain knowledge. Evaluated on multiple Kaggle benchmark datasets with synthetically injected errors, our approach significantly improves accuracy of downstream classification and regression models. However, it remains challenging for cross-row distributional errors (e.g., global trend deviations). Our core contribution is the first LLM-driven, fully automated, executable, and feedback-enhanced tabular data cleaning paradigm.

Contemporary data engineering pipelines rely heavily on manual feature engineering, which is labor-intensive and lacks generalizability. To address this, we propose ADEPT—a novel framework that introduces text embedding entropy as a principled information-theoretic measure for time series. ADEPT jointly optimizes textual time-series representation, pre-trained text embedding models, and the variational information bottleneck (VIB) to enable end-to-end automated time-series data engineering. It eliminates handcrafted feature design by directly extracting high-entropy representations from raw textualized time series while suppressing embedding variance to enhance robustness. Evaluated across diverse benchmarks in healthcare, finance, scientific computing, and industrial IoT, ADEPT consistently outperforms state-of-the-art methods. It demonstrates exceptional robustness to missing values, formatting errors, and irregular timestamps, significantly improving both data science efficiency and scalability.

This work addresses the lack of traceability in existing domain-specific fine-tuning approaches, which often leads to blind and inefficient data augmentation. The authors propose a “programming with data” paradigm that treats structured knowledge representations as a unified foundation for both training and evaluation, drawing an analogy to software development: training data serve as source code, model training as compilation, evaluation as unit testing, and data refinement as debugging. This framework enables precise, concept- and reasoning-chain–oriented model repair through structured knowledge extraction, test-driven data engineering, concept-level gap analysis, and diagnosis of broken reasoning chains. Validated across 16 disciplines, the approach significantly enhances model performance without compromising general capabilities, and the authors release an open-source knowledge base, evaluation suite, and training corpora to support reproducibility and further research.

This work addresses the gap between theory and practice in error detection and cleaning for tabular data by proposing and implementing an interactive web-based demonstration system that integrates error modeling, injection, and intelligent cleaning. For the first time, the system unifies machine learning–based data cleaning methods with dependency-aware error generation models within a visual platform, enabling users to upload tables, inject realistic errors, and interactively experience the cleaning process alongside interpretable explanations of the underlying mechanisms. By synergizing machine learning, database technologies, and web development, the project enhances intuitive understanding of error repair principles and provides a publicly accessible demo platform (https://cured.demo.calgo-lab.de/) that serves as an effective validation tool for both research and practical applications in data cleaning.

Pattern-based file access remains fundamental yet lacks systematic methodological support in contemporary computational research. Method: This paper systematically integrates Python’s `glob` module into data science and AI workflows, establishing an efficient, scalable framework for pattern-driven file discovery, filtering, and data loading. By tightly coupling `glob` with mainstream libraries—including `pandas`, `scikit-learn`, and others—we implement an end-to-end automated pipeline spanning raw file ingestion, structured organization, and construction of AI-ready datasets. Contribution/Results: We establish `glob` as a foundational, reproducible methodology for scientific data engineering—the first such formalization in the literature—thereby addressing critical gaps in both practice and documentation of pattern matching across interdisciplinary computational research. The approach significantly enhances automation, reproducibility, and maintainability of data processing pipelines, enabling robust, transparent, and scalable data engineering practices in research and production settings.

This study addresses the challenges of accurately querying structured data and extracting information from unstructured clinical text in electronic health records (EHRs). To this end, the authors propose a unified framework that integrates large language models (LLMs) with retrieval-augmented generation (RAG): LLMs are employed to execute structured queries (e.g., Pandas operations), while RAG enhances information extraction from unstructured clinical narratives. The work introduces an innovative automatic evaluation pipeline based on synthetically generated question-answer pairs, combining exact match metrics, semantic similarity scores, and human assessments. Evaluated on a subset of MIMIC-III, the approach demonstrates improved semantic accuracy and task adaptability, offering clinical data science a flexible and reliable tool for automated reasoning and evaluation.

This work addresses the growing bottlenecks in large language model (LLM) training caused by overreliance on scaling data volume, which faces limitations in data availability, cost, and efficiency. The authors propose a hierarchical data management framework spanning the entire training lifecycle (L0–L4), which uniquely integrates LLMs directly into the data processing pipeline. This framework establishes a five-tier hierarchy transforming raw corpora into verifiable knowledge, enabling model-guided data filtering, quality scoring, and dynamic allocation. It supports heterogeneous learning objectives across pretraining, mid-training, and alignment stages, significantly enhancing both training efficiency and model performance. To foster co-evolution of data and models, the project releases the hierarchical dataset and accompanying tools as open-source resources.