expert curation

In data science, task-oriented dataset discovery faces challenges including implicit user preferences, opaque search spaces, and ambiguous relevance criteria—leading to inefficient query iteration. This paper proposes an AI-augmented dataset search framework that explicitly models user intent via AI-driven query reformulation; enables fine-grained content understanding through joint column- and row-level semantic analysis; and introduces a task-driven relevance metric with dynamically generated, interpretable feedback. The framework supports users in progressively constructing a cognitive model of the search space, thereby enhancing both interpretability and interactive efficiency. Experiments demonstrate that, compared to keyword-based and conventional semantic search methods, our approach improves structured exploration efficiency by 37% and reduces query iterations by 42%. Its effectiveness and generalizability are validated across multiple real-world data science tasks.

Contemporary dataset papers frequently suffer from limited originality, insufficient diversity, inadequate quality control, and poor transparency regarding construction methodologies; existing datasheets are largely descriptive and lack quantifiable evaluation criteria or enforceable accountability mechanisms. Method: We propose DataRubrics, the first rubric-based framework for structured data quality assessment, integrating LLM-as-a-judge (e.g., GPT-4) with synthetic data techniques to enable automated, reproducible, and standardized quality scoring for both human- and model-generated datasets. Contribution/Results: The framework delivers an open-source evaluation toolkit (github.com/datarubrics/datarubrics), facilitating collaborative, measurable data review by reviewers and authors alike. It significantly enhances rigor, transparency, and trustworthiness in data-centric research through objective, interpretable, and auditable quality metrics.

Dataset quality defects—such as missing documentation, incorrect labels, and ethical risks—are pervasive in open platforms yet resistant to detection by rule-based scripts, necessitating intelligent, automated identification methods. Method: We introduce the first LLM-agent benchmark for discovering real-world dataset quality issues, covering 221 empirically validated cases across eight platforms. It uniquely evaluates agents’ ability to autonomously detect latent defects without prior prompting. We propose an automated evaluation framework powered by GPT-4o, achieving high agreement with human experts (Cohen’s κ = 0.89), and ensure benchmark reliability via multi-source real-data sampling and expert annotation. Contribution/Results: Experiments reveal that even the state-of-the-art Curator agent detects only ~30% of defects, underscoring task difficulty. All benchmark data, code, and evaluation tools are publicly released to advance intelligent data governance.

Addressing the challenge of constructing high-quality, domain-specific annotated data—often costly and labor-intensive—this paper proposes a few-shot-driven synthetic data generation paradigm. Given only a small set of user-provided examples, the method retrieves semantically relevant real-world text from large-scale web corpora and leverages instruction-tuned large language models (LLMs) to automatically generate well-formatted, task-specific synthetic training data. It is the first approach to synergistically integrate corpus retrieval with LLM-based augmentation, enabling zero human annotation, domain adaptability, and efficient few-shot generalization. Empirical evaluation across biomedical, medical, and commonsense question answering (QA), as well as summarization tasks, demonstrates that models trained on the generated data achieve a 46-point preference score improvement over human-annotated baselines in summarization, while QA models match or surpass the performance of general-purpose foundation models.

This work proposes a novel dataset discovery framework that leverages citation contexts from scientific papers to better capture the semantic intent behind research queries, addressing the limitations of existing dataset search engines that rely primarily on metadata and keyword matching and consequently suffer from low recall. By treating citation context as the core signal—combined with large-scale context extraction, large language model–guided pattern recognition, and provenance-preserving entity resolution—the approach significantly reduces dependence on incomplete or inconsistent metadata. Evaluated on eight computer science queries, the method achieves an average normalized recall of 47.47% (peaking at 81.82%), substantially outperforming Google Dataset Search and DataCite Commons. The framework’s novelty and practical utility have been affirmed by domain experts across multiple disciplines.

This work addresses the high cost of machine learning benchmarking by proposing a systematic framework to efficiently select small, representative subsets of datasets while preserving model ranking stability. The study presents the first comprehensive evaluation of various dataset selection strategies—including clustering, A/D-optimal experimental designs, random baselines, and a greedy farthest-first (FAFI) approach—on rank fidelity. It derives a theoretical upper bound on Spearman rank correlation error for FAFI and integrates bootstrap aggregation to yield statistically rigorous confidence intervals for comparing strategy performance. Empirical results demonstrate that as few as five datasets suffice to achieve 0.95 rank correlation in time series classification, significantly outperforming random selection in NLP tasks, though gains are limited in recommendation systems.

This study addresses the challenge of low-quality metadata that hinders dataset discoverability and reuse, particularly in the context of large language model (LLM)-generated descriptions lacking empirical guidance on context selection and its impact on quality. Building a literature-based framework for description quality assessment, the authors conduct systematic ablation experiments across 252 real-world CSV datasets. They uncover a previously unreported “table-structure penalty” phenomenon: relying solely on table structure significantly degrades narrative quality. While representative data samples aid semantic grounding, they do not improve overall human-rated quality. The work further reveals that different LLMs exhibit consistent descriptive styles. Through LLM-as-a-judge evaluations, semantic attribute analysis, and large-scale experimentation, the study offers key recommendations for LLM-assisted data publishing: concise, relevant context yields better results than redundant input, and table structure should be used cautiously as a basis for generation.

This work addresses the challenge that domain experts face in translating natural language descriptions of data quality requirements into executable analyses, a process often hindered by reliance on data engineers, resulting in inefficiency and high technical barriers. To overcome this, the paper proposes a no-code, model-driven pipeline that leverages a QPM metamodel to define domain-specific quality analysis templates. Coupled with the Constrainify toolchain, it automatically transforms natural language requirements into executable and reusable analytical logic. By integrating model-driven engineering, metamodeling, and no-code web technologies, the approach significantly reduces dependency on technical expertise, enabling efficient, reproducible, and semantically aligned data quality assessments. This advancement enhances both the accessibility and automation of data quality analysis for non-technical domain practitioners.

This study addresses the challenge that domain experts, due to limited query language proficiency, often struggle to independently conduct context-specific data quality analyses and must rely on technical specialists, resulting in inefficient workflows. To overcome this limitation, the paper proposes the Quality Pattern Model (QPM) framework—a novel, template-based mechanism that is agnostic to both database technologies and application domains, enabling non-technical users to autonomously define data quality analysis logic. Leveraging a model-driven approach, the authors implement QPM prototypes over XML, RDF, and Neo4j. Experimental results demonstrate that QPM’s expressiveness matches or exceeds that of mainstream query languages while significantly enhancing domain experts’ analytical autonomy. The framework’s effectiveness has been validated in the cultural heritage domain.

This study addresses the longstanding reliance on manual labor in cataloging digital collections—a process hindered by low efficiency and high costs. The authors systematically evaluate the performance of various artificial intelligence models in automated cataloging tasks, employing both quantitative metrics and qualitative analysis to comprehensively assess accuracy, robustness, and applicability. Their investigation identifies the model architectures best suited for cataloging scenarios and distills a set of transferable, cross-domain principles for AI-driven cataloging. These findings offer both theoretical grounding and practical guidance for cultural heritage institutions seeking to advance their digital transformation through intelligent technologies.