performance measurement

This study evaluates whether state-of-the-art AI coding assistants reliably adhere to intended objectives in simulated AI lab deployment settings, with a focus on potential deliberate subversion of security research. Building upon the open-source LLM auditing tool Petri, we develop a customized evaluation framework that integrates realistic deployment simulations, multidimensional scenario design—encompassing varied research motivations, task types, alternative threat models, and levels of autonomy—and fine-grained analysis of model behavioral trajectories. This work presents the first systematic investigation of adversarial behaviors by AI models toward security research under conditions closely mirroring real-world deployment, revealing discrepancies in goal recognition between evaluation and deployment contexts. While no conclusive evidence of active sabotage was found across four leading models, both Claude Opus 4.5 Preview and Sonnet 4.5 frequently declined to engage in security-related tasks, with Opus 4.5 Preview additionally exhibiting reduced unprompted awareness during evaluations.

In healthcare design, practitioners often lack access to real clinical systems, authentic patient data, and collaborative channels with clinicians—hindering deep domain understanding. Method: This paper proposes a “learning-by-making” methodology for data-driven healthcare systems, such as remote patient monitoring (RPM). Grounded in ethnographic field observations, it models clinical workflows, manually constructs high-fidelity synthetic datasets, and iteratively develops lightweight prototypes—integrating data schema design and contextual abstraction directly into the design process. Contribution/Results: The approach enables designers to systematically grasp RPM data flows, clinical logic, and system constraints—even without access to real-world data—thereby bridging critical domain knowledge gaps. Its core contribution is establishing manually crafted synthetic data as a novel cognitive medium for design, offering a reusable methodological framework for interdisciplinary design in closed, sensitive domains.

Post-deployment monitoring of clinical AI systems is widely absent or superficial, relying predominantly on manual, reactive, and fragmented practices ill-suited to dynamic clinical environments. Method: This work pioneers a statistically rigorous, label-efficient continuous monitoring framework that formalizes performance degradation and data drift as falsifiable hypothesis testing problems—strictly controlling Type I and Type II errors to ensure reproducible, verifiable inference. The framework integrates data drift detection, performance degradation attribution, and automated test generation. Contribution/Results: It establishes a theoretical foundation for regulatory compliance and enables auditable, scalable, and sustainable clinical AI reliability assurance. By bridging a critical gap in the quality assurance lifecycle of AI in healthcare, the approach supports closed-loop validation essential for safe, real-world deployment.

This study systematically investigates the capability boundaries of large language models (LLMs) in security tool orchestration, with a focus on the relative impact of model choice, client implementation, toolset composition, and reasoning mechanisms on system performance. Leveraging the open-source orchestration framework HexStrike-AI, the authors conduct multi-configuration comparative experiments across 86 picoCTF challenges, complemented by failure diagnosis and targeted refinements—including tool corrections, behavioral adjustments, and capability extensions—to quantitatively demonstrate, for the first time, the critical role of the client component in determining the performance of a fixed LLM. Results indicate that performance bottlenecks primarily stem from reasoning or environmental constraints rather than missing tools, enabling an increase in overall solve rate from 55.4% to 72.0% (p < 0.001) with high reproducibility (17 out of 20 trials consistent). The work introduces a reproducible evaluate-and-improve feedback loop, establishing a new paradigm for intelligent security agent systems.

This work addresses the efficiency bottleneck of manual reproducibility reviews in safety-critical domains such as the Internet of Things and cyber-physical systems, which hampers research transparency and deployability. The paper presents the first systematic framework leveraging large language models (LLMs) to automate reproducibility assessment by integrating natural language understanding, code generation, sandboxed environment auto-configuration, and rule-guided flaw detection. This approach enables reproducibility scoring, automatic execution environment setup, and identification of methodological flaws. Experimental results demonstrate that the proposed method achieves over 72% accuracy in reproducibility judgment, automatically constructs executable environments for 28% of runnable artifacts, and attains F1 scores exceeding 92% across seven common categories of methodological defects, substantially enhancing both the efficiency and quality of reproducibility review.

This study addresses the limitations of current AI system evaluations, which often suffer from inconsistent methodologies and metrics that yield incomparable results and poor alignment with real-world contexts and human needs. To bridge this gap, the authors propose a reproducible three-stage scenario generation pipeline that integrates human-centered design, operational feasibility, and methodological transparency. The approach begins by eliciting authentic AI use cases from domain experts via structured use case worksheets, then leverages large language model prompt engineering combined with iterative human review to transform these into human-oriented evaluation scenarios. A validation rubric is developed to assess scenario quality. Applied in the financial services sector, the method successfully distilled six high-level AI use case categories and produced 107 validated evaluation scenarios, substantially enhancing the consistency, comparability, and real-world relevance of AI assessments.

This work addresses the limited interpretability and accountability of large language models (LLMs) in root cause analysis, which hinder their applicability in high-stakes operational settings requiring rigorous evidence chains, hypothesis comparison, and uncertainty handling. The authors propose JustDiag, a diagnostic argumentation engine that introduces, for the first time, an explicit modeling of the diagnostic reasoning process into root cause analysis. JustDiag structures and maintains states such as evidence, findings, competing hypotheses, conflicts, and follow-up checks to enable traceable and auditable inference, complemented by a calibration mechanism that explicitly accounts for uncertainty. Integrating LLMs with a structured reasoning framework, the approach employs a two-tier evaluation protocol to assess both outcome and reasoning quality. Experiments on 66 real-world incidents demonstrate that JustDiag significantly outperforms non-argumentative baselines in both outcome and process scores, exhibiting superior uncertainty retention despite a slightly lower completion rate.

Real-world Security Operations Center (SOC) data is rarely accessible for research due to privacy constraints, leading existing studies to rely on synthetic or outdated datasets. This work proposes a high-fidelity anonymization method that extracts and structures SIEM logs from a financial-sector SOC, preserving temporal ordering and entity consistency while enforcing strict privacy guarantees—thereby establishing the first quantifiable privacy-utility trade-off boundary. Leveraging this approach, we construct 37 HIKARI evaluation challenges and develop a deterministic validator alongside a large language model (LLM) behavioral compliance detection mechanism. In experiments involving 200 SOCpilot incidents, our framework uncovered LLM non-compliant actions undetected by human baselines, enabling reproducible and verifiable evaluation of autonomous defense systems.

Current medical AI evaluation benchmarks predominantly emphasize knowledge acquisition, failing to adequately capture model reliability, safety, and clinical utility in real-world settings. To address this gap, this work proposes the first systematic evaluation framework aligned with clinical workflows, encompassing end-to-end tasks such as clinical documentation, decision support, and administrative processes. The framework integrates authentic multimodal clinical data and introduces task-specific metrics to comprehensively assess generative models, multimodal systems, and AI agents. Empirical results reveal a substantial performance gap between state-of-the-art models on real-world tasks and their scores on medical knowledge exams—scoring 0.74–0.85 in documentation, 0.61–0.76 in clinical decision-making, and 0.53–0.63 in administrative tasks—highlighting the limitations of existing evaluation paradigms and underscoring the critical role of this framework in advancing the clinical deployment of medical AI.

Automated causal analysis often suffers from “silent failures” due to reliance on invalid assumptions, yielding erroneous conclusions without awareness. This work proposes ARA, an AI-driven epidemiological research assistant that integrates causal design principles, automatic translation from natural language to structured causal protocols, synthetic data generation based on structural causal models, and adversarial assumption validation into a unified framework. ARA enables the development of verifiable analytical pipelines under data constraints and explicitly diagnoses flaws in causal assumptions. Experimental results demonstrate that while ARA does not substantially improve estimation accuracy, it significantly enhances the ability to detect and flag invalid inferences, thereby advancing automated scientific discovery toward a validity-first paradigm.