frechet inception distance

This paper identifies a systematic misalignment between generic generative evaluation metrics—such as Fréchet Inception Distance (FID)—and downstream task performance (e.g., classification or segmentation) in retinal image synthesis. Method: Through systematic experiments across multimodal retinal datasets (fundus photography and OCT), we empirically analyze the correlation between FID (and its variants) and actual gains in downstream model performance. Contribution/Results: We provide the first empirical evidence that FID scores fail to predict whether synthetic data meaningfully improve downstream model accuracy. To address this, we propose a “task-driven evaluation” paradigm, advocating direct assessment via target downstream task performance—replacing proxy metrics reliant on ImageNet-pretrained features. Our findings are robustly replicated across multiple public retinal image benchmarks, offering both methodological insight and practical guidance for evaluating biomedical image generation.

This work systematically investigates the discrepancy between Fréchet Inception Distance (FID) and human perception, demonstrating that low FID scores do not necessarily correspond to high-quality generated outputs. The study identifies the geometric structure of the reference dataset—particularly its distribution density and effective rank—as the primary cause of FID’s unreliability. Through empirical analyses involving precision-recall decomposition, multiple feature spaces, and ablation studies on distance metrics, the authors reveal that FID exhibits reasonable behavior on concentrated datasets but becomes misleading on dispersed ones. Experiments across six datasets confirm that the geometric properties of the reference data critically influence the fidelity of distribution-based evaluation metrics. The findings advocate for interpreting such metrics in conjunction with the underlying dataset geometry, offering a more reliable foundation for evaluating generative models.

This study investigates the impact of image similarity—quantified by the Fréchet Inception Distance (FID)—on the efficacy of synthetic data augmentation for few-shot diabetic macular edema (DME) retinal image segmentation. We employ Progressive Growing GAN (PGGAN) to generate synthetic retinal images and systematically analyze the quantitative relationship between FID scores and U-Net segmentation performance. Results demonstrate that lower FID values—indicating higher visual fidelity of synthetic images to real data—correlate strongly with more significant and stable improvements in segmentation accuracy. Crucially, synthetic-data augmentation exhibits a markedly distinct performance trajectory compared to conventional geometric augmentation, revealing a strong nonlinear dependence of augmentation effectiveness on image similarity. The key contribution is the first empirical validation in medical few-shot segmentation that FID serves as a reliable predictor of synthetic data quality; moreover, we establish FID < 25 as a critical threshold for enhancing model generalization.

Existing image generation evaluation metrics—such as the Fréchet Inception Distance (FID)—exhibit limited sensitivity to fine-grained visual discrepancies, higher-order distributional moments, and tail behavior, and lack statistical rigor. To address these limitations, we propose the Embedding Characteristic Score (ECS), the first metric to formally establish theoretical connections between distributional assessment and higher-order moments as well as tail characteristics, thereby overcoming FID’s insensitivity to non-Gaussian tail mismatches. ECS is grounded in kernel embedding theory and characteristic function analysis, and is validated through Monte Carlo simulations and empirical image experiments across both synthetic and real-world benchmarks. Results demonstrate that ECS significantly outperforms FID and other state-of-the-art metrics in detecting distributional shifts, mode collapse, and tail misalignment. It provides a statistically principled, interpretable, and robust evaluation framework for generative models.

提出基于小波包变换的Fréchet Wavelet Distance(FWD)指标，解决现有生成图像评价指标对特定生成器和数据集的偏见问题，通过计算小波包系数空间的Fréchet距离，实现领域无关且更可解释的质量评估。

This work addresses critical limitations of existing generative model evaluation metrics—such as the Fréchet Inception Distance (FID)—in sample complexity, computational efficiency, and robustness against adversarial attacks. The authors propose the Monge Inception Distance (MIND), which computes a one-dimensional optimal transport average of Inception features via sliced Wasserstein distance, thereby circumventing the need for high-dimensional mean and covariance estimation. MIND achieves evaluation performance comparable to FID with only 1k–5k samples, offering an order-of-magnitude improvement in sample efficiency and two orders of magnitude faster computation. Moreover, it demonstrates significantly enhanced robustness against adversarial manipulations such as moment matching. While maintaining high correlation with FID, MIND exhibits superior discriminative power, substantially facilitating efficient generative model development and iteration.

Current evaluations of generative models often report a single FID score, neglecting the randomness inherent in both training and sampling processes, which undermines reliability and reproducibility. This work is the first to model FID as a random variable jointly determined by training and sampling seeds. By training hundreds of SiT models on ImageNet at 256×256 resolution, we systematically quantify the sources of FID variance. Our analysis reveals that training seeds dominate FID variation—contributing approximately 3.2 times more than sampling seeds—and that optimal classifier-free guidance reduces FID dispersion by nearly half. Moreover, FID differences with a coefficient of variation (CoV) below 1.3% should be deemed statistically insignificant. Building on these insights, we propose a new evaluation paradigm incorporating multi-seed error bars and optimal guidance, substantially enhancing robustness and reproducibility in generative model assessment.

This study addresses the limitations of conventional image quality metrics—such as FID, SSIM, KID, IS, and LPIPS—in evaluating synthetic remote sensing imagery, demonstrating their poor alignment with both human perception and downstream task performance. By generating Earth observation images using deep generative models, the work systematically compares automatic metrics against human judgments and semantic segmentation accuracy for land cover classification. The findings reveal a significant misalignment: semantic-preserving perturbations substantially alter automatic scores without affecting human interpretability, and low-scoring synthetic samples can enhance segmentation performance when used in mixed training regimes. Moreover, metrics based on ImageNet feature spaces prove unreliable for geospatial data, underscoring the necessity of grounding synthetic data evaluation in task-specific performance and human assessment rather than generic image fidelity measures.

Existing image quality assessment metrics are constrained by closed vocabularies and rigid parametric assumptions, limiting their ability to accurately evaluate high-quality generated images. This work proposes the APEX framework, which introduces— for the first time—the parameter-free, assumption-free sliced Wasserstein distance into image quality evaluation. By leveraging dual foundation models, CLIP and DINOv2, APEX extracts open-vocabulary features and measures distributional similarity through projected analysis in a high-dimensional embedding space. This approach overcomes the expressivity and generalization limitations of conventional metrics, demonstrating exceptional stability and robustness across diverse visual degradation scenarios, cross-dataset evaluations, and out-of-domain data.

Existing text-to-image generation evaluation metrics struggle to distinguish between foreground subjects and background due to their global image processing, leading to inaccurate assessments of concept fidelity and prompt adherence. This work proposes MaSC, the first spatially decomposed evaluation paradigm that leverages external foreground masks to decouple assessment into subject-specific concept fidelity and background-related prompt following. Built upon a frozen SigLIP2 SO400M-NaFlex model, MaSC incorporates masked maximum cosine matching, background-pooled embeddings, and subject-removed prompt contrastive scoring. Experiments demonstrate that MaSC achieves a Krippendorff’s α of 0.471 for concept fidelity on DreamBench++ and an identity recognition AUC of 0.992 on ORIDa, significantly outperforming CLIP-T baselines and exhibiting stronger alignment with human perception.