🤖 AI Summary
Existing evaluation metrics for generative models suffer from poor robustness and ill-defined fidelity-diversity trade-offs—structural deficiencies that hinder the practical deployment of synthetic data.
Method: This paper introduces, for the first time, normative principles for synthetic data evaluation and an interpretable “sanity-check” suite. Through theoretical analysis and controlled experiments—including anomaly injection, distribution collapse, and scale perturbation—we systematically assess 12 mainstream metrics across three canonical failure modes.
Contribution/Results: We demonstrate that all widely adopted metrics fail to meet practical reliability requirements. Based on these findings, we formulate a misuse warning checklist and a safety-guided usage protocol. Our work shifts the evaluation paradigm from “model-centric” to “metric-centric,” establishing a methodological foundation for trustworthy synthetic data assessment.
📝 Abstract
Any method's development and practical application is limited by our ability to measure its reliability. The popularity of generative modeling emphasizes the importance of good synthetic data metrics. Unfortunately, previous works have found many failure cases in current metrics, for example lack of outlier robustness and unclear lower and upper bounds. We propose a list of desiderata for synthetic data metrics, and a suite of sanity checks: carefully chosen simple experiments that aim to detect specific and known generative modeling failure modes. Based on these desiderata and the results of our checks, we arrive at our position: all current generative fidelity and diversity metrics are flawed. This significantly hinders practical use of synthetic data. Our aim is to convince the research community to spend more effort in developing metrics, instead of models. Additionally, through analyzing how current metrics fail, we provide practitioners with guidelines on how these metrics should (not) be used.