artifact removal

The core challenge in data quality monitoring lies in error provenance—specifically, identifying the underlying mechanisms that generate errors—a problem largely overlooked by existing work, which seldom models such mechanisms explicitly. This paper focuses on errors arising from intrinsic dependencies within data and proposes MechDetect, the first method to systematically extend missing-data mechanism detection to diverse error types—including outliers, inconsistencies, and format violations. Leveraging joint statistical modeling and supervised learning, MechDetect simultaneously models tabular data and their error masks to automatically determine whether observed errors stem from inherent characteristics of the original data. Extensive experiments across multiple benchmark datasets demonstrate that MechDetect significantly outperforms state-of-the-art baselines in accurately diagnosing error-generation mechanisms. By providing mechanistic interpretability, it establishes a theoretical foundation and practical framework for explainable data repair.

In industrial quality inspection, anomaly detection suffers from poor robustness due to high noise levels and sparse defective samples. To address this, we propose Iterative Refinement of Pseudo-labels (IRP), a self-supervised method that alternately evaluates sample credibility and removes misleading instances under feature-space consistency constraints—effectively purifying the training set dynamically without human annotations and generating high-fidelity self-supervised signals. IRP introduces the novel paradigm of “iterative data refinement,” significantly enhancing model robustness against label noise and cross-domain generalization capability. Evaluated on KSDD2 and MVTec AD benchmarks, IRP consistently outperforms existing unsupervised and self-supervised methods. Notably, under high-noise conditions, it achieves substantial improvements in detection accuracy and reduces false positive rates by over 25%.

In highly regulated domains such as finance, data quality control (QC) is often fragmented into isolated preprocessing steps, undermining end-to-end trustworthy AI pipelines. To address this, we propose the first AI-driven DataOps framework that embeds QC as a system-level core component. Our framework deeply integrates rule-based engines, statistical analysis, and custom AI-powered anomaly detection across the entire data lifecycle—from ingestion and transformation to model deployment—enabling dynamic remediation, policy-configurable workflows, and end-to-end auditability. Technically, it unifies data profiling, stream processing, cloud-native storage interfaces, and a proprietary AI detection module. Evaluated in a real-world financial production environment, the framework achieves significantly improved anomaly recall, reduces manual intervention by 42%, and ensures audit completeness and full data traceability under high-throughput conditions, fully satisfying regulatory compliance requirements.

Industrial prediction and soft sensing often fail due to field data suffering from bias, latency, or seemingly plausible yet unreliable measurements. This work proposes a large language model (LLM)-guided Measurement Credibility Correction (MCC) method that, for the first time, leverages semantic information from process documentation to construct an external reference—requiring neither numerical correlations, fault labels, nor explicit process equations—for lightweight pre-inference correction. MCC translates document semantics into reference signals compatible with numerical models and integrates them at the front end of the prediction pipeline. Evaluated on multiple real-world industrial tasks, MCC reduces average relative MAE by 30.7% on authentic test data and by 80.3% under controlled contamination, while adding only 0.5–2.0k online parameters and incurring a maximum inference latency of 0.089 ms per step.

This study investigates the joint impact of data missingness and noise on machine learning performance. We systematically quantify trade-offs among data quality, volume, and imputation strategies across two representative scenarios: NLP supervised learning (BERT) and traffic signal control via reinforcement learning (PPO). Methodologically, we propose a novel “Performance Degradation Index Model under Data Corruption” and identify that only 30% of critical data governs overall model performance. We introduce the concepts of “Imputation Advantage Angle” and “Imputation Disadvantage Edge,” and— for the first time—categorize learning tasks into noise-sensitive versus noise-insensitive classes. Results show that noise degrades performance more severely than missingness; imputation efficacy critically depends on alignment between imputation accuracy and data corruption rate; and merely scaling data volume only mitigates—not eliminates—corruption effects, with diminishing marginal returns intensifying as corruption worsens.

This work addresses the long-standing limitation in automatic defect detection for magnetic flux leakage (MFL) imaging—namely, the absence of large-scale public datasets and standardized benchmarks. To bridge this gap, we introduce PipeMFL-240K, the first large-scale object detection dataset and benchmark specifically designed for pipeline MFL pseudocolor images. It comprises over 240,000 images and 190,000 high-quality bounding box annotations across 12 defect categories, faithfully capturing real-world challenges such as long-tailed class distribution, small object scales, and high intra-class variability. Leveraging this dataset, we conduct a systematic evaluation of state-of-the-art detection models, revealing their performance bottlenecks in MFL scenarios and establishing a reliable, reproducible platform to foster future algorithmic innovation and pipeline integrity assessment.

Existing approaches treat data quality assessment and machine learning systems as disjoint components, hindering dynamic, real-time coordination in production environments. This paper proposes the first end-to-end, quality-driven framework tailored for industrial MLOps, achieving the first closed-loop integration of data quality evaluation and model inference. The framework introduces theoretically grounded yet engineering-practical mechanisms: dynamic distribution drift detection, adaptive multi-dimensional quality metrics, a lightweight inference pipeline, and configurable quality thresholding. Evaluated on an industrial steelmaking ESR vacuum pump process, it achieves a model R² of 94%—a 12-percentage-point improvement—and reduces prediction latency by 75%, enabling millisecond-level quality-aware decision-making.

Existing trajectory anomaly detection and cleaning tools lack systematic, standardized evaluation, hindering fair comparison and practical deployment. Method: We propose a unified taxonomy categorizing methods into five classes—statistical, sliding-window, clustering, graph-based, and heuristic—and introduce a reproducible ground-truth generation mechanism tailored to real-world trajectory scenarios. This forms the first standardized evaluation framework for trajectory anomaly detection. Contribution/Results: We conduct comprehensive efficiency and accuracy benchmarking across ten mainstream open-source tools on diverse real-world trajectory datasets, analyzing performance degradation under distinct anomaly types (e.g., positional drift, sampling noise, semantic inconsistency). Our empirical study yields an evidence-based tool selection guideline, significantly enhancing comparability and practicality of trajectory preprocessing methods. The framework establishes a foundational benchmark for both academic research and industrial applications in trajectory data quality management.

This work addresses the lack of fine-grained, sample-level traceability across multi-stage machine learning pipelines in existing transparency mechanisms. The authors propose FG-Trac, a model-agnostic framework that enables verifiable, per-sample tracking without modifying model architectures or training objectives. By integrating cryptographic commitments, capturing sample lifecycle events, and computing contribution scores based on training checkpoints, FG-Trac reconstructs a complete and tamper-proof history of data usage spanning both preprocessing and training phases. Experimental results demonstrate that FG-Trac efficiently provides auditable evidence of individual sample propagation paths in convolutional neural networks and multimodal graph learning tasks, all while preserving predictive performance.

This work addresses the propagation of label errors in data validation, which can severely compromise the reliability of downstream query results. To quantify the impact of such errors and identify high-risk tuples whose uncertainty may be exacerbated by validation, the authors propose Maximum Error Score (MES)—a data-distribution-agnostic metric. Building on MES, they design MESReduce, an interactive validation optimization algorithm that adaptively guides the verification process by efficiently computing MES and incorporating feedback from external validators. Experimental evaluation on both real-world and synthetic datasets demonstrates that MESReduce significantly reduces the maximum error score and effectively enhances validation accuracy.