literature survey

This work addresses the frequent neglect of sampling strategy design and generalizability in software engineering research, which often undermines the representativeness of empirical findings. To remedy this, the paper introduces a domain-specific language (DSL) that explicitly models complex sampling workflows over code repositories through composable sampling operators, enabling—for the first time—formal specification and reasoning about the generalizability of sampling strategies. Implemented as a fluent Python API, the DSL is integrated with a statistical metric system to quantitatively assess the external validity of sampled datasets. The authors demonstrate the expressiveness and practical utility of their approach by reconstructing and formalizing the sampling procedures from multiple Mining Software Repositories (MSR) studies, thereby validating the framework’s capacity to capture real-world methodological diversity.

Current evidence synthesis workflows adopt a flat, single-layer structure, leading to redundant method reconstruction across projects, conceptual drift, and unstable cross-project inference. To address these issues, we propose a three-tiered, inheritance-based meta-architecture comprising the *Methodological Law Layer*, the *Domain Abstraction Layer*, and the *Project Implementation Layer*. This design enforces hierarchical separation, explicit routing rules, and contamination control to ensure construct clarity and reproducibility. We introduce the first formal inheritance governance framework, defining three ontologically distinct entities—*Grandparent* (methodological laws), *Parent* (domain abstractions), and *Child* (project-specific implementations)—to enable systematic, lineage-aware methodological evolution and robust cross-project reasoning. Empirical evaluation demonstrates significant improvements in cross-project methodological consistency and replication rates. The architecture supports the construction of a scalable, multi-project evidence ecosystem and provides a sustainable, inheritable methodological infrastructure for long-term research programs.

Experimental reproducibility in Empirical Software Engineering (ESE) is hindered by a fundamental disconnect between idealized methodological assumptions—e.g., standardized protocols and controlled conditions—and researchers’ actual experimental practices. Method: We conducted a two-year ethnographic study involving participant observation, in-depth interviews, and content analysis of experimental artifacts across diverse ESE research teams. Contribution/Results: We identify four critical dimensions—activity diversity, role distribution, conceptual granularity, and domain perspective—in which real-world experimentation systematically deviates from textbook models. Based on these findings, we propose the first high-fidelity conceptual and process model grounded in empirical research practice, explicitly capturing the “practice gap” underlying irreproducibility. This model provides foundational evidence and design principles for developing next-generation reproducibility-support tools, methodological guidelines, and evaluation frameworks in ESE.

This work addresses the challenge of efficiently and objectively evaluating the novelty of scholarly submissions in peer review, particularly amidst the rapidly expanding volume of scientific literature. To this end, we propose an intelligent agent system powered by large language models that implements a four-stage pipeline—contribution extraction, semantic retrieval, hierarchical classification coupled with fine-grained full-text comparison, and evidence synthesis—to deliver an end-to-end, traceable, and evidence-based novelty assessment grounded in actual published works. This approach effectively mitigates hallucination risks inherent in large language models. Deployed on over 500 submissions to ICLR 2026, our method accurately identifies relevant prior work omitted by authors, significantly enhancing the fairness, consistency, and interpretability of peer reviews. All evaluation reports have been publicly released.

Existing research ideation tools emphasize breadth-oriented idea generation but lack support for iterative refinement, elaboration, and evaluation—hindering literature-grounded, deep-reading–driven conceptual evolution. Method: We propose the first literature-driven interactive research ideation system, integrating a composable “idea element” canvas model with a multi-dimensional (problem/solution/evaluation/contribution) co-evolution mechanism. Our approach innovatively incorporates LLM-powered literature-aware feedback generation, graph-structured idea modeling, and interactive multi-path variant exploration. Contribution/Results: Experiments demonstrate a 42% increase in user-generated idea output and significantly enhanced detail elaboration. Seven researchers successfully applied the system across the full ideation pipeline—from initial topic conception to paper outline revision—validating its efficacy in supporting deep, iterative, literature-informed research design.

This work addresses the inefficiency and steep learning curve researchers often encounter when trying to map academic papers to their corresponding implementation code. To bridge this gap, the authors propose an automated tool powered by large language models (LLMs) that achieves cross-modal semantic alignment between scholarly texts and source code for the first time. By integrating program analysis techniques, the method automatically identifies code segments that implement specific research ideas described in a paper and generates high-quality traceability mappings. This approach substantially reduces the manual effort required for alignment, enhances the comprehensibility of research software, and improves reproducibility. Preliminary experiments demonstrate the tool’s practicality and effectiveness in real-world scenarios.

This work addresses the lack of systematic evaluation of scientific agents in multi-turn interaction, multimodal evidence integration, and long-term memory utilization. To bridge this gap, we propose the first multi-turn, multimodal benchmark grounded in real-world scientific workflows, requiring agents to perform cross-paper retrieval, align textual and visual evidence, and answer complex questions demanding cross-document comparison and fusion of multiple figures within a short time horizon. The benchmark introduces a fine-grained, process-level evaluation framework that emphasizes active retrieval, multi-source fusion, and memory-augmented reasoning. Experimental results reveal that even state-of-the-art models achieve only 29.23% accuracy on the challenging test set, highlighting substantial room for improvement and providing the community with a reproducible evaluation platform.

This study addresses the tendency of systematic reviews to overgeneralize by overlooking fine-grained characteristics of included studies, thereby obscuring inter-study relationships and gaps in the literature. To mitigate this limitation, the authors propose an interactive evidence mapping approach that integrates large language models, topic modeling, and visualization techniques to automatically extract themes from heterogeneous review data and construct a dynamically explorable knowledge map. Validation through a scoping review on pedagogical agents in K–12 education demonstrates that this method transcends the constraints of traditional static summaries, substantially enhancing review transparency, effectively uncovering latent patterns and research gaps, and strengthening exploratory analytical capabilities.

This work addresses a critical yet often overlooked source of evaluation bias in database research: performance variability arising from non-deterministic compiler behavior across different builds, even when the source code remains unchanged. To systematically identify and mitigate this threat to reproducibility, the paper introduces the Multi-Version Experimental Evaluation (MVEE) framework. MVEE automatically detects “build anomalies” through assembly-level analysis, materializes them as distinct method versions, and incorporates these variants into a unified evaluation protocol. By explicitly accounting for implicit compilation-induced differences, MVEE substantially enhances the reliability and expressiveness of experimental results, ensuring that performance comparisons reflect genuine algorithmic characteristics rather than artifacts of the build process. This approach strengthens the scientific rigor and credibility of empirical studies in database systems research.

Current mechanisms struggle to verify whether revisions to scientific manuscripts substantively address peer reviewers’ concerns with supporting textual evidence. This work proposes AutoSupervision, a novel framework that leverages transparent peer review records from 56,000 papers in Nature Communications to construct a closed-loop evaluation system powered by large language models (e.g., GPT-5.5). The system automatically identifies reviewer concerns, assesses the effectiveness of author revisions, and locates supporting evidence within the revised text. Experimental results show that large language models achieve strong performance in identifying reviewer concerns (F1 = 0.754), yet evidence-based verification remains challenging, with the best current model attaining only an F1 score of 0.501. This study establishes a verifiable, evidence-driven paradigm for automated assessment of scientific manuscript revisions.