Transparency in Healthcare AI: Testing European Regulatory Provisions against Users' Transparency Needs

2025-05-21

AI Summary

This study assesses whether the transparency requirements for medical AI Instructions for Use (IFUs) under the EU AI Act (Regulation 2024/1689) align with the heterogeneous needs of diverse stakeholders. Method: A large-scale, multi-group online survey was conducted via Qualtrics targeting healthcare managers, clinicians, patients, and IT specialists, integrating demand-relevance scoring and cross-role mapping consistency analysis. Contribution/Results: The study provides the first empirical evidence of systematic differences in transparency expectations across these four user groups and reveals a significant misalignment between current IFU structural conventions and users’ critical informational needs. Based on these findings, we propose a “localized, role-adapted” IFU design framework, specifying actionable optimization pathways. The results support regulatory compliance implementation while enhancing trustworthy human-AI interaction in clinical practice, offering an evidence-based foundation and methodological reference for global governance of medical AI transparency.

Technology Category

• Humans and AI: User Experience and Usability
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Security and Privacy: Data transparency and provenance
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

Artificial Intelligence (AI) plays an essential role in healthcare and is pervasively incorporated into medical software and equipment. In the European Union, healthcare is a high-risk application domain for AI, and providers must prepare Instructions for Use (IFU) according to the European regulation 2024/1689 (AI Act). To this regulation, the principle of transparency is cardinal and requires the IFU to be clear and relevant to the users. This study tests whether these latter requirements are satisfied by the IFU structure. A survey was administered online via the Qualtrics platform to four types of direct stakeholders, i.e., managers (N = 238), healthcare professionals (N = 115), patients (N = 229), and Information Technology experts (N = 230). The participants rated the relevance of a set of transparency needs and indicated the IFU section addressing them. The results reveal differentiated priorities across stakeholders and a troubled mapping of transparency needs onto the IFU structure. Recommendations to build a locally meaningful IFU are derived.

Problem

• Evaluating if EU AI Act's transparency requirements meet user needs
• Assessing clarity and relevance of healthcare AI Instructions for Use
• Identifying stakeholder-specific transparency priorities in healthcare AI

Innovation

• Testing IFU structure against transparency needs
• Surveying stakeholders via Qualtrics platform
• Mapping transparency needs to IFU sections