format conversion

Existing ETL pipelines heavily rely on manual, context-sensitive design of transformation logic, resulting in poor generalizability and low reusability. To address this, we propose an example-driven autonomous ETL framework: given user-provided target data examples, it constructs a paired-sample-based planning engine that automatically infers and synthesizes high-fidelity, context-adapted data transformation programs. Integrated with modular ETL components and runtime monitoring, the framework enables end-to-end automation for multi-format, multi-structured, and multi-scale data processing. Experiments across 14 real-world, cross-domain datasets demonstrate that our approach substantially reduces human intervention while achieving high-precision transformations (average F1 score of 0.92), strong generalization across diverse schemas and formats, and practical engineering deployability.

Large-scale text cleaning, filtering, and formatting for large language model (LLM) data engineering face high technical barriers and suffer from fragmented, non-integrated tooling. Method: This paper introduces the first low-code, block-based open-source ETL framework specifically designed for LLM data engineering. It features a modular, configuration-driven pipeline architecture with pluggable processor interfaces, enabling rapid integration of custom data processing logic; supports both CLI and programmatic API invocation to balance usability and flexibility. Contribution/Results: Experiments demonstrate efficient automated preprocessing of TB-scale corpora, significantly reducing LLM data preparation overhead. The framework is fully open-sourced, accompanied by tutorial videos and comprehensive documentation, thereby advancing standardization and community-driven development in LLM data engineering.

This study addresses the challenges of maintaining consistency across heterogeneous schema languages—such as JSON Schema, XSD, and SHACL—during multilingual data model evolution, where fragmented converters, variable quality, and information loss impede reliable interoperability. The work proposes a novel approach that models schema languages and black-box converters as nodes and directed edges in a graph, enabling composable and evaluable conversion path orchestration. By integrating graph-based search, quality-aware ranking (combining agent-assisted and human evaluation), and failure backtracking, the method supports automated, reproducible cross-language schema transformation. The resulting open-source toolchain, Schema Conversion Orchestrator, integrated into the MetaConfigurator platform, successfully produced valid outputs for 43 out of 60 real-world tasks and precisely identified missing ecosystem components in the remaining 17, thereby delineating the current boundaries of schema conversion capabilities.

This work addresses data quality, multilingual coverage, and regulatory compliance challenges in training large language models (LLMs) for the OpenGPT-X initiative. Methodologically, it introduces a novel “dual-track” data processing paradigm: lightweight filtering for curated datasets and aggressive filtering combined with MinHash/LSH-based deduplication for large-scale web corpora—fully aligned with EU regulations such as the GDPR. The pipeline integrates fastText-based language identification, hybrid rule-and-statistics filtering, a learned quality scoring model, and end-to-end metadata provenance tracking. Its primary contribution is the construction of the first high-quality, EU-compliant multilingual corpus for LLM training, explicitly designed for public-sector applications. Empirical evaluation demonstrates substantial improvements in model robustness, transparency, and auditability—particularly in government and public service use cases—while ensuring legal and ethical adherence across 24 official EU languages.

To address static copyright management, poor provenance tracking, and delayed license updates in AI model training, this paper proposes IBis—a blockchain-based framework introducing the first on-chain/off-chain coordination mechanism tailored to the dynamic training trajectory of AI models. IBis leverages Daml smart contracts deployed on the Canton platform to enable on-chain registration, full-lifecycle provenance tracing, and fine-grained dynamic license management for datasets, models, and associated permissions. It supports verifiable retraining and automated license renewal in multi-stakeholder settings. Unlike static NFT-based copyright solutions, IBis achieves low latency and high throughput under realistic workloads (hundreds of users, thousands of assets), demonstrating industrial-grade feasibility and horizontal scalability. As the first infrastructure for decentralized AI training, IBis enables real-time compliance verification and iterative license evolution.

This work proposes the first fully large language model–driven, end-to-end data integration framework that eliminates the need for manual configuration, which traditionally incurs high costs and low efficiency. The system autonomously generates a complete integration pipeline encompassing schema mapping, value normalization, entity matching, and conflict resolution without human intervention. Evaluated on three real-world domains—gaming, music, and enterprise data—the GPT-5.2–based framework achieves integration performance comparable to or surpassing that of handcrafted systems. Notably, it accomplishes this at a remarkably low cost of approximately $10 per execution, substantially reducing human labor and operational overhead.

This work addresses the lack of end-to-end public benchmarks for data integration by introducing MaDI-Bench, the first comprehensive benchmark that spans the entire relational table integration pipeline—including schema matching, value normalization, entity blocking, entity matching, and data fusion. To mitigate benchmark saturation, MaDI-Bench incorporates a set of foundational cross-domain tasks along with a mechanism for generating extensible task variants. The benchmark supports both step-wise and end-to-end evaluation of system performance, validated through diverse pipelines ranging from manual and optimal combinations to large language model (LLM)-based approaches. All resources are publicly released to foster reproducible and holistic assessment of data integration systems.

This study addresses a critical yet previously unrecognized issue in knowledge graph construction: the coupling between tabular serialization formats and schema constraints, which significantly degrades both factual coverage and graph fidelity—particularly in country-year statistical tables, where it induces entity inflation or extraction failure. The authors formally identify and name this phenomenon “format-constraint coupling,” introduce a direct graph access evaluation paradigm, and release CSVFidelity-Bench, a benchmark comprising diverse table types and gold-standard facts. Through factorial experiments, bootstrap confidence intervals, token ablation studies, and multi-LLM comparisons, they uncover significant positive coupling effects in four out of six datasets (peak effect size +1.180). Direct graph access reveals a quality gap as large as 47.6 percentage points (p<0.0001), substantially exceeding that of standard retrieval-based approaches.

Existing data pipelines often suffer from weak governance, leading to delayed schema validation, inconsistent cross-language execution, and misalignment with business semantics. This work proposes treating data contracts as types, leveraging the “everything-as-code” paradigm to inject schema annotations—encompassing column types, constraints, documentation, and lineage—into input and output tables within a lakehouse architecture via multi-language SDKs. These annotations are parsed across multiple phases of the execution lifecycle, deeply integrating data contracts into the type system. The approach enables both deterministic and non-deterministic reasoning over data flows across languages and execution engines, significantly enhancing the reliability of production data pipelines and ensuring consistent interoperability across systems.

This study addresses the persistent challenges of accuracy and robustness in natural language to SQL (NL2SQL) translation under complex query scenarios. The authors systematically evaluate the combined effects of multiple optimization strategies—including the NatSQL intermediate representation, synthetic data preprocessing and fine-tuning, and a novel SQL re-ranking model—using SmBoP and RASAT as backbone architectures. Through ablation studies and Shapley value analysis, they quantitatively assess, for the first time, the interaction effects among these components, revealing that their performance gains are not merely additive. The results demonstrate that non-trivial combinations of these techniques yield significant improvements on benchmarks such as Spider, underscoring the critical role of synergistic interactions among system components.