🤖 AI Summary
While existing late-interaction retrieval models offer interpretability, they lack systematic evaluation of whether their understanding of clinical concepts is stable, reusable, and context-sensitive. This work proposes a knowledge-guided latent space alignment method that aligns token embeddings from ColBERT to a reference latent space constructed from a clinical knowledge graph and expert-defined concept similarity constraints. By doing so, document encodings become verifiable evidence of semantic understanding. This approach is the first to integrate external clinical knowledge into a late-interaction architecture, enabling precise identification of biomedical concept misunderstandings without requiring extensive diagnostic queries. It further provides clear guidance for targeted data curation, substantially enhancing the reliability and maintainability of retrieval systems.
📝 Abstract
Reliable biomedical and clinical retrieval requires more than strong ranking performance: it requires a practical way to find systematic model failures and curate the training evidence needed to correct them. Late-interaction models such as ColBERT provide a first solution thanks to the interpretable token-level interaction scores they expose between document and query tokens. Yet this interpretability is shallow: it explains a particular document--query pairwise score, but does not reveal whether the model has learned a clinical concept in a stable, reusable, and context-sensitive way across diverse expressions. As a result, these scores provide limited support for diagnosing misunderstandings, identifying irreasonably distant biomedical concepts, or deciding what additional data or feedback is needed to address this. In this short position paper, we propose Diagnosable ColBERT, a framework that aligns ColBERT token embeddings to a reference latent space grounded in clinical knowledge and expert-provided conceptual similarity constraints. This alignment turns document encodings into inspectable evidence of what the model appears to understand, enabling more direct error diagnosis and more principled data curation without relying on large batteries of diagnostic queries.
