evaluate optical performance

Photonic integrated circuit (PIC) design relies heavily on repetitive, error-prone manual coding with limited automation. To address this, we introduce PICBench—the first LLM evaluation benchmark specifically targeting PIC design automation, centered on netlist generation across diverse tasks ranging from elementary photonic components to complex circuits. PICBench establishes a novel dual-dimension automated evaluation framework assessing both syntactic correctness and functional correctness; it is the first to integrate open-source photonic simulators (SAX/NumPy) for end-to-end functional verification, enabling reproducible and scalable benchmarking. Comprehensive evaluation of state-of-the-art LLMs—including GPT and Llama series—reveals critical limitations in physical consistency and functional accuracy. PICBench thus provides a foundational benchmark and actionable insights for advancing intelligent photonic EDA tools.

This work addresses the problem of implementation drift in evolving distributed systems, where runtime behavior gradually deviates from the original design. To tackle this issue, the paper proposes a design conformance assessment method based on distributed tracing data. It introduces, for the first time in the domain of distributed systems, conformance checking techniques from process mining, leveraging runtime traces collected via the OpenTelemetry standard and automatically comparing them against behavioral models defined at design time to quantify their alignment. The key contribution lies in establishing persistent, monitorable conformance metrics that enable continuous, automated evaluation of deviations between system implementation and design. This approach is readily applicable to modern distributed systems widely adopting OpenTelemetry for observability.

Scientific software selection frequently suffers from non-reproducible benchmarks due to multi-library, multi-metric evaluation and dynamic evolution—such as the introduction of new algorithms or modifications to test cases and evaluation criteria. This paper addresses numerical integration over arbitrary 2D/3D domains with implicit or parameterized boundaries (cut-cell quadrature), proposing the first automated benchmarking framework that systematically integrates CI/CD engineering practices into scientific computing workflows. The framework unifies GitHub Actions, Docker, Python-based scheduling, Jupyter-based report generation, and semantically versioned result archiving. It supports automated configuration, execution, visualization, and historical result comparison. It achieves >90% automation for benchmark tasks and regression detection; reduces integration time for new libraries or algorithms by 70%; and enables precise attribution of performance deviations to specific code commits. The framework significantly enhances reliability, reproducibility, and evolutionary adaptability in scientific software evaluation.

This study addresses the lack of a systematic overview of open-source software energy measurement tools, which hinders energy-aware software design and tool selection. From a mining software repositories (MSR) perspective, the authors employ qualitative content analysis to screen and categorize 585 GitHub projects, identifying 24 high-quality open-source energy measurement tools. The work systematically characterizes these tools in terms of architectural design, measurement granularity—spanning from CPU-level to process, container, and AI workload levels—and their capabilities for carbon emission estimation. By elucidating evolutionary trends in tool development, this research provides software architects with a structured foundation and practical guidance for informed tool selection in energy-efficient software engineering.

Silicon photonic integrated circuits lack systematic design-for-testability methodologies, making it challenging to efficiently detect manufacturing variations and functional faults. This work introduces, for the first time, a design-for-testability framework tailored to this domain, proposing a generic test architecture that leverages dedicated test access and fault-detection circuitry to achieve high-coverage validation of optical signal power and phase. The approach accommodates complex topologies, including those with feedback loops, and is co-designed with silicon photonic device modeling, electromagnetic simulation, and manufacturing variation analysis. Simulation results on feedforward photonic neural networks and feedback-based photonic logic circuits demonstrate that the proposed architecture effectively identifies photonic signal anomalies, significantly enhancing functional verification capability.

This study addresses a critical gap in quantum software research: the absence of a systematic auditing mechanism for empirically grounded comparative claims, which has led to a pervasive “instantiation gap” characterized by insufficient evidentiary support. To bridge this gap, the authors propose CLAIMSTAB-QC, the first source-bound auditing framework tailored to empirical comparisons in quantum software. By integrating claim modeling, audit scope delimitation, evidence boundary identification, and directional classification, the framework enables precise validation of comparative assertions against original source materials. An evaluation across 455 claims from 119 papers reveals that only eight claims possessed sufficient matched evidence for direct auditing; among these, two were confirmed, four lacked adequate support, and two were contradicted—highlighting substantial deficiencies in the empirical rigor of current quantum software studies.

This work addresses the lack of a standardized observability framework in quantum networks, which hinders effective fault diagnosis and adaptive control. It proposes the first multidimensional performance metric system tailored for quantum networks, encompassing key parameters such as entanglement fidelity, quantum bit error rate, dark count rate, and timing jitter, while integrating environmental sensor data. Building on this foundation, the authors design and implement a non-intrusive, integrable real-time monitoring prototype, which has been deployed and validated at Oak Ridge National Laboratory. The system enables real-time data acquisition, performance alerting, and dynamic feedback, thereby establishing a critical observability infrastructure for quantum software-defined networking and autonomous control.