output comparison

This study addresses a critical limitation in traditional reproducible research, where sharing only code and results fails to expose the implicit assumptions, expectations, and premises underlying an analyst’s reasoning—thereby hindering thorough evaluation of analytical quality. To overcome this, the paper proposes a formal modeling framework that explicitly translates the analyst’s tacit reasoning process into structured logical representations, statically capturing the construction logic of the analysis. This approach enables systematic scrutiny of the analytical chain of reasoning, assumption sensitivity, and conclusion robustness—even in the absence of the original data. Empirical validation on representative data analysis tasks demonstrates the framework’s effectiveness, achieving both logical visualization and data-free static assessment of analytical integrity.

Agentic data science systems often produce deceptively optimistic conclusions that are difficult to detect and lack robust mechanisms for reliability validation. This work addresses this challenge by introducing falsifiability constraints into the domain for the first time, leveraging the Predictability-Computability-Stability (PCS) framework to propose a lightweight dual-check mechanism. The approach evaluates an agent’s ability to distinguish signal from noise through controlled perturbations, thereby quantifying the stability of its conclusions. Experiments across 11 real-world datasets reveal that in 6 of them, the affirmative conclusions generated by agentic systems lack empirical support, and their self-reported confidence levels exhibit significant misalignment with actual stability. These findings underscore both the necessity and effectiveness of the proposed method.

This work addresses the unreliability of Chain-of-Thought (CoT) monitors in detecting undesirable behaviors—such as test-time exploitation—often stemming from insufficient information extraction or poor approximation of the monitoring function. For the first time, it formalizes CoT monitorability from an information-theoretic perspective, establishing that non-zero mutual information between the CoT and the output is necessary but insufficient for effective monitoring. The study identifies two key error sources: information gaps and steering errors. To mitigate these, it proposes a novel label-free joint optimization framework that combines conditional mutual information maximization with oracle-guided reinforcement training to systematically enhance monitor performance. Experiments demonstrate that this approach significantly improves monitoring accuracy across diverse settings, effectively suppresses CoT degradation, and alleviates reward hacking even under imperfect reward signals.

This study addresses the challenge of detecting information leakage solely from a model’s predictive outputs, without access to training code, external data, or domain knowledge. Framed within decision theory, the approach models leakage diagnosis as a functional of predictive risk and outcome distribution, linking proper scoring rules with decision curve analysis via threshold-weighted associations to enable detection without prior assumptions. The work introduces a novel tripartite classification of information leakage—miscalibration, generalized calibration, and determinism—and theoretically establishes that generalized calibration leakage is fundamentally unidentifiable, whereas near-deterministic subgroups can be efficiently detected. Empirical validation on UK Biobank demonstrates detection of temporal-window comorbidity leakage down to Δc*≈0.007 in under one second, while also revealing inherent structural limitations of purely output-driven leakage detection.

Existing AI-generated content (AIGC) fact-checking tools predominantly rely on black-box binary classification or regression models, suffering from poor interpretability, limited evidence diversity, and minimal user interactivity. Method: We propose the first user-driven, fine-grained fact verification framework that decomposes long texts into atomic claims, integrates heterogeneous multi-source evidence (e.g., knowledge bases, web pages, documents), and employs cross-source evidence fusion with an interpretable reasoning model to produce claim-level confidence scores and natural-language explanations—supporting multi-hop provenance tracing and dynamic user feedback. Contribution/Results: Our framework breaks from conventional paradigms by enabling transparent, traceable, evidence-diverse, and human-AI collaborative verification. Experiments demonstrate significant improvements in user verification efficiency (+37%) and trust (+42%), establishing a novel paradigm for trustworthy AIGC interaction.

This study addresses the challenges posed by rapid evolution in digital forensic systems and tools, which induces drift in evidentiary behaviors and tool outputs, thereby undermining result reproducibility and trustworthiness. To mitigate this, the authors propose a test-driven forensic methodology that introduces state-transition testing for causal attribution, encoding forensic expectations as executable specifications. The approach integrates virtual machine environments with computer vision–guided GUI automation to simulate authentic user interactions and verify system state changes. An open web platform is developed to facilitate sharing and replication of experiments. The method’s efficacy is demonstrated through five case studies, including a regression analysis across 25 versions of Autopsy, which uncovered numerous undocumented, substantial changes in its reporting output.

This work addresses the problem of resource wastage in random forest classifiers caused by semantic errors—such as those stemming from data imbalance—that often remain undetected until after model training. To mitigate this, the authors propose a data-driven static analysis method that operates without access to the original dataset. By modeling machine learning scripts as directed acyclic graphs and integrating formal API contracts with aggregated data properties, the approach enables early detection of silent semantic errors even in confidential environments. The resulting open-source tool, dille, demonstrates high efficacy on real-world Kaggle notebooks, identifying such errors in 12%–18% of random forest scripts with 91% precision and sub-second analysis overhead.

This work addresses a common yet critical issue in machine learning code: semantic errors arising from mismatches between data properties and model assumptions—such as applying scale-sensitive models to unnormalized data—which traditional debugging approaches can only detect after training, resulting in inefficiency. To enable early and automatic error detection, the authors propose a novel data-aware static analysis method that integrates dataflow and control-flow analysis with API specifications, thereby incorporating data semantics directly into the static analysis framework for the first time. Evaluation on real-world machine learning notebooks demonstrates that the approach effectively identifies subtle semantic bugs that conventional techniques fail to catch, highlighting its practical utility and methodological innovation.

This work addresses the challenge in distributed tracing diagnostics where anomalies often differ from normal behavior across multiple unknown dimensions—such as structure, timing, critical paths, and semantics—making them difficult to capture effectively through a single interface. To tackle this, the authors propose Contrast, a novel system that decouples trace representation from comparison semantics and introduces a composable Trace Projection Object (TPO) to uniformly encode multidimensional features. This design enables dynamic construction of arbitrary comparison sets at query time. Contrast further integrates two complementary interfaces: interactive visualization and large language model–generated explanations, facilitating flexible and in-depth root cause analysis. Experimental evaluation demonstrates that Contrast significantly improves both multidimensional comparison efficiency and diagnostic accuracy on the DeathStarBench benchmark and in Uber’s production environment.

This work addresses the prevalent yet often undetectable issue of logical inconsistency between reasoning and final answers in chain-of-thought (CoT) outputs generated by current AI systems during safety evaluations. The study is the first to formally distinguish between reasoning consistency and faithfulness, introducing a taxonomy encompassing six distinct types of inconsistency. To enable post-hoc detection without modifying model generation, the authors propose InspectScout—a reusable scanning method grounded in formal definitions, supported by a human-annotated benchmark, and implemented via an automated detection algorithm integrated into the inspect_evals framework. Experiments demonstrate that reasoning inconsistencies are widespread across four mainstream models and three safety-related tasks, and can be reliably identified; moreover, the patterns of such inconsistencies exhibit systematic variation across models.