What You See is (Usually) What You Get: Multimodal Prototype Networks that Abstain from Expensive Modalities

2025-11-24

AI Summary

Species detection faces two key bottlenecks: poor interpretability of multimodal models and heavy reliance on costly, invasive genetic data. To address these, we propose an interpretable, cost-aware multimodal prototype network built upon the ProtoPNet framework. It jointly leverages visual and genetic modalities, incorporating a gated weighting mechanism for dynamic, instance-wise modality fusion, and a rejection module that actively abstains from querying expensive genetic data under low-confidence predictions. Furthermore, cost-sensitive learning is integrated to optimize modality selection policies. Experiments demonstrate that our method achieves accuracy comparable to full-modality baselines while substantially reducing genetic data usage—up to 62% in some settings. This validates the feasibility of image-centric, high-accuracy, low-cost fine-grained species identification. The approach establishes a new paradigm for ecological monitoring that simultaneously ensures model transparency, operational efficiency, and practical deployability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Species detection is important for monitoring the health of ecosystems and identifying invasive species, serving a crucial role in guiding conservation efforts. Multimodal neural networks have seen increasing use for identifying species to help automate this task, but they have two major drawbacks. First, their black-box nature prevents the interpretability of their decision making process. Second, collecting genetic data is often expensive and requires invasive procedures, often necessitating researchers to capture or kill the target specimen. We address both of these problems by extending prototype networks (ProtoPNets), which are a popular and interpretable alternative to traditional neural networks, to the multimodal, cost-aware setting. We ensemble prototypes from each modality, using an associated weight to determine how much a given prediction relies on each modality. We further introduce methods to identify cases for which we do not need the expensive genetic information to make confident predictions. We demonstrate that our approach can intelligently allocate expensive genetic data for fine-grained distinctions while using abundant image data for clearer visual classifications and achieving comparable accuracy to models that consistently use both modalities.

Problem

• Developing interpretable multimodal networks for species detection
• Reducing reliance on expensive genetic data collection
• Maintaining accuracy while selectively using costly modalities

Innovation

• Multimodal prototype networks ensemble modality-specific prototypes
• Weighted modality reliance enables interpretable cost-aware predictions
• Selective genetic data allocation maintains accuracy while reducing costs