Rethinking Data Quality for AI-Driven Systems: Evidence from Practitioner Interviews

2026-09-25

AI Summary

This study addresses the lack of empirical evidence in data quality management for AI systems, where traditional perspectives struggle with model attribution and compliance challenges. Employing reflexive thematic analysis through in-depth interviews with 16 practitioners, this work examines the engineering and organizational dimensions of data quality in AI-driven systems, revealing emergent characteristics including traceability, circularity, and legitimacy. It introduces a novel conceptual framework termed “lifecycle assurance” that integrates fragmented machine learning research agendas and establishes evidence-generation mechanisms supporting specific AI claims. Furthermore, the study identifies six overarching themes and five trust-influencing conditions, offering practice-based, engineering-oriented guidance for managing data quality in AI systems.

Abstract

Data quality research has usually treated data as an input that is stored, processed, and validated. In AI-driven software-intensive systems, data also shapes model behavior, evaluation, and lawful use. Empirical evidence remains limited on how practitioners define, assess, and manage quality under these conditions. We interviewed 16 practitioners from nine organizations and analyzed the transcripts using reflexive thematic analysis and developed six themes from participants' accounts. In AI systems, traceability shifted from modular debugging to attributing model behavior, while using models as quality assessors introduced circularity. Agent context and memory became data objects, and synthetic and pseudo-labeled data made authenticity a quality concern. In foundation-model development, lawfulness became a gate for training data, while representativeness was judged through coverage of situations in which the system must behave safely. Prior ML research examines many of these problems separately. Our study provides a practitioner-grounded account of how they are encountered together as an engineering and organizational concern. We also interpret five recurring conditions as helping explain how the themes relate to reduced trust in data and AI outcomes. We synthesize these findings through lifecycle assurance: a conceptual framing focused on producing evidence that data can support a specific AI claim when its influence may be embedded in model behavior, model-based judgments, or agent actions.

Problem

• data quality
• AI-driven systems
• practitioner perspectives
• traceability
• foundation models

Innovation

• data quality
• AI-driven systems
• lifecycle assurance
• foundation models
• traceability