CANDOR: Chance-Calibrated Discordance in Frozen Foundation Encoders

2026-07-20

AI Summary

Existing evaluation methods struggle to accurately assess the semantic support provided by frozen foundation encoders for downstream tasks due to interference from class imbalance. This work proposes CANDOR—a label-symmetric inconsistency metric rigorously calibrated to an opportunity level of 0.5—that evaluates the separation quality between same-class and different-class samples in feature space without requiring a trained prediction head. By analyzing nearest neighbors and feature geometry using equally sized positive and negative sample banks, CANDOR was validated across 22 encoders, 20 datasets, and over 600,000 images. Results reveal that while most encoders are not “blind,” their representational capacity is generally limited; notably, CANDOR can unbiasedly and prospectively identify insufficient task-specific support from these encoders.

Abstract

Frozen encoders are chosen by how well a lightweight head reads a finding from their features, not whether the geometry separates it. Nearest-neighbor discordance does, but with unequal banks the opposite-label neighbor wins on density, not geometry, so prevalence alone makes an uninformed encoder look blind. We introduce CANDOR, a discordance measure whose equal-size banks are symmetric under a label swap, fixing its chance level at exactly one half. Across 22 encoders, 20 datasets from 7 domains, and 605,443 images, this correction reverses the conclusion. Collapse falls below chance almost everywhere, so no encoder is blind, yet all are weak: the best chest model reads pneumothorax at 84.5 AUROC and still places 18.4% of those positives nearer an opposite-label film than its own kind in the same hospital. The same encoder that resolves bird species at 4.5 leaves chest findings at 42.8 and glaucoma at 49.8, at chance and worse than random weights. Such a case caps the normalized margin of any Lipschitz head, yet some head among eleven is correct on all but 2.8% of cases where one head misses 35.9%: the deficit is selection, not information. Erasure retention is associated with collapse; we detect no association with the objective, scale, recency, or size of the finding. Because the chance level is fixed, CANDOR can be read before any head is trained, flagging which findings a frozen encoder supports poorly.

Problem

• frozen encoders
• discordance
• class imbalance
• feature geometry
• evaluation bias

Innovation

• CANDOR
• frozen encoder
• chance-calibrated discordance
• representation collapse
• nearest-neighbor geometry