design auditability frameworks

AI systems frequently lack auditability—their capacity to undergo independent, lifecycle-wide evaluation against ethical, legal, and technical standards—due to fragmented regulatory frameworks and the absence of systematic auditing practices. Method: This study proposes an “audit-embedded governance” framework that proactively integrates auditing requirements into AI development workflows. It establishes a holistic methodology comprising standardized compliance documentation, dynamic risk assessment protocols, multi-tiered governance structures, and interoperable auditing tools. The approach emphasizes cross-stakeholder collaboration and socio-technical co-design, while advocating for international regulatory alignment and mutual recognition. Contribution/Results: The framework clarifies actionable pathways for scaling AI audits across organizational and jurisdictional boundaries. It delivers a practical, implementation-oriented guide for building trustworthy, transparent, and accountable AI governance systems, alongside concrete policy recommendations for regulators, developers, and auditors.

This work addresses the lack of traceable and tamper-resistant transparency mechanisms in large language models (LLMs) deployed in high-stakes decision-making contexts, which undermines accountability. To bridge this gap, the paper introduces the first LLM lifecycle auditing framework that integrates technical provenance with governance records. It proposes a reference architecture enabling cross-organizational traceability and implements a lightweight, open-source Python-based auditing layer. By leveraging append-only logs, event emitters, structured metadata, and an auditor interface, the system seamlessly integrates into existing LLM workflows with minimal intrusiveness. This design ensures complete, tamper-evident traceability across critical stages—including training, deployment, and monitoring—thereby facilitating robust accountability and responsibility attribution throughout the model’s lifecycle.

Current AI-assisted scientific writing lacks auditable generation processes and mechanisms for accountability, undermining the verifiability of research credibility and compliance. This work proposes a novel auditing paradigm embedded directly within the production workflow, enforcing end-to-end traceability, immutability, and third-party reproducibility of AI involvement through preregistered blind-spot indicator cards, sealed execution environments, and automated gatekeeping intercepts. Core technical components include Git-sealed lineage anchoring, hash-bound provenance tracking, red-flag interception protocols, cross-model role isolation, and programmatic assembly. In experimental validation, one project was automatically terminated when preregistered confirmatory tests triggered a No-Go decision. An open-source toolkit is released to enable independent recomputation of all core audit metrics by third parties.

This work addresses the challenge of reconciling task-level verification and regulatory traceability within high-velocity AI-assisted engineering workflows. The authors propose an “infinite loop” framework that integrates agile iteration with V-model validation, embedding independent verification and compliance auditing into every development cycle through a multi-agent AI architecture. The system automatically generates audit-ready documentation and incorporates critical human-in-the-loop approval gates. By natively embedding compliance capabilities into the development process, the approach achieves 100% requirement-level verification and enables trustworthy delivery with minimal human intervention. In a hardware-in-the-loop case study, the system attained full requirement pass rates with an average of only six human prompts per cycle, demonstrating a projected cost reduction of 10–50× compared to conventional methods.

Current AI auditing tools predominantly focus on model performance evaluation, failing to support end-to-end accountability practices—including harm identification, evidence construction, stakeholder engagement, and intervention advocacy. Method: We conducted in-depth interviews with 35 practitioners and systematically crawled, cataloged, and coded 435 auditing tools to develop a需求–capability mapping framework and an ecosystem gap diagnostic model. Contribution/Results: Our analysis reveals systematic deficiencies across four critical capabilities: traceability, multi-stakeholder participation, evidentiary chain generation, and intervention support—constituting the first empirical diagnosis of such gaps. Building on these findings, we propose the “AI Accountability Infrastructure” paradigm, shifting beyond narrow assessment-centric design toward cross-stage coordination and multi-role adaptability. The study delivers an empirically grounded, prioritized roadmap for designing next-generation, accountability-oriented AI auditing tools.

Industrial research agents often generate experimental trajectories containing invalid or incomplete information, rendering them unreliable for direct decision-making. This work proposes an evidence-oriented framework that automatically transforms such trajectories into structured evidence through a context-isolated generate–verify–repair pipeline. The approach introduces intervention-level claim categorization—distinguishing actionable repairs, diagnostic safeguards, and retained discoveries—and incorporates end-to-end provenance tracking to enable claim scoping and auditability. Experimental results demonstrate that the resulting candidate solutions outperform existing baselines. Audits further reveal that trajectory evolution is non-monotonic, and that applicability assessment constitutes a key performance bottleneck for the controller.

Current perturbation-based construct validity audits are highly sensitive to implementation details, yielding conclusions that lack transparency and reliability. This work proposes a self-audit framework that systematically identifies and formalizes five classes of audit failure modes (F1–F5). A case study encompassing two open-source instruction-tuned models and five safety benchmarks reveals that none of the audited units satisfy confirmatory criteria, exposing systemic vulnerabilities in prevailing practices. To address this, the paper introduces a six-point due diligence gating mechanism that establishes actionable standards for disclosing and retaining high-assurance audit evidence, thereby substantially enhancing the credibility and reproducibility of auditing outcomes.

This study addresses the challenge in IT auditing where evidence from heterogeneous organizations is fragmented and compliance with security and regulatory controls must be assessed based on semantic adequacy rather than keyword matching, hindering automation. To tackle this, the work proposes the first system integrating Retrieval-Augmented Generation (RAG) with a multi-agent collaboration framework. The system orchestrates evidence retrieval, evaluation generation, adversarial challenge of assertions, and resolution of disagreements to produce interpretable audit recommendations that include citations, reasoning, gap analysis, and remediation guidance. Experimental validation under the ISO/IEC 27001 standard demonstrates that the approach effectively supports control interpretation and audit preparation, significantly improving efficiency. Nevertheless, human oversight remains necessary to calibrate judgments of evidentiary sufficiency.

This study addresses the limitations of current AI auditing practices, which predominantly focus on individual models while overlooking integration risks arising from interactions among system components and between systems and their environments. Through a scoping review and reflexive thematic analysis of 58 studies, the work systematically codes existing literature to delineate, for the first time, three distinct domains of AI integration auditing: inter-component, system–environment, and multi-system. It further introduces domain-specific evaluation dimensions—compatibility, completeness, and oversight—that capture unique aspects of integrated AI systems. The findings reveal that current auditing practices remain fragmented and nascent, underscoring the critical role of accessible information and resource support in effective audit design. The paper calls for novel auditing frameworks capable of spanning components, environments, and systems to enable systematic exploration, identification, coordination, and standardization of integration-related risks.