🤖 AI Summary
The uncritical adoption of “standard” benchmark datasets often obscures semantic mismatches between their labels and the target task’s intended meaning, undermining model trustworthiness. Method: We propose the first dataset suitability assessment framework integrating grounded theory with visual hypothesis testing, prioritizing task-oriented data quality evaluation through qualitative coding and empirical case studies (20 Newsgroups and MNIST). Contribution/Results: Our analysis reveals severe label misalignment in 20 Newsgroups—its high test accuracy stems from spurious correlations rather than genuine semantic understanding—whereas MNIST labels are empirically validated as reliable. The study demonstrates that dataset standardization does not imply task-specific suitability; rigorous, task-grounded re-evaluation of data quality is essential. Our framework provides a reproducible, interpretable methodology for assessing dataset trustworthiness in context-sensitive machine learning applications.
📝 Abstract
Standard datasets are frequently used to train and evaluate Machine Learning models. However, the assumed standardness of these datasets leads to a lack of in-depth discussion on how their labels match the derived categories for the respective use case, which we demonstrate by reviewing recent literature that employs standard datasets. We find that the standardness of the datasets seems to cloud their actual coherency and applicability, thus impeding the trust in Machine Learning models trained on these datasets. Therefore, we argue against the uncritical use of standard datasets and advocate for their critical examination instead. For this, we suggest to use Grounded Theory in combination with Hypotheses Testing through Visualization as methods to evaluate the match between use case, derived categories, and labels. We exemplify this approach by applying it to the 20 Newsgroups dataset and the MNIST dataset, both considered standard datasets in their respective domain. The results show that the labels of the 20 Newsgroups dataset are imprecise, which implies that neither a Machine Learning model can learn a meaningful abstraction of derived categories nor one can draw conclusions from achieving high accuracy on this dataset. For the MNIST dataset, we demonstrate that the labels can be confirmed to be defined well. We conclude that also for datasets that are considered to be standard, quality and suitability have to be assessed in order to learn meaningful abstractions and, thus, improve trust in Machine Learning models.