From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification

2026-02-28

AI Summary

This study addresses the limitations of existing pet identification systems, which suffer from small-scale datasets and reliance on a single visual modality, leading to suboptimal performance in lost-pet retrieval tasks. To overcome these challenges, the authors construct a large-scale pet dataset comprising 1.9 million images and present the first systematic exploration of multimodal fusion methods in this domain. They propose an enhanced framework that incorporates synthetic text descriptions as semantic priors, leveraging the SigLIP2-Giant visual encoder and the E5-Small-v2 text encoder. Through comprehensive evaluation of fusion strategies—ranging from feature concatenation to adaptive gating—the study demonstrates that gated fusion significantly sharpens decision boundaries. Experimental results show a Top-1 accuracy of 84.28% and an equal error rate of 0.0422, representing an 11% improvement over the best single-modality baseline.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Automated animal identification is a practical task for reuniting lost pets with their owners, yet current systems often struggle due to limited dataset scale and reliance on unimodal visual cues. This study introduces a multimodal verification framework that enhances visual features with semantic identity priors derived from synthetic textual descriptions. We constructed a massive training corpus of 1.9 million photographs covering 695,091~unique animals to support this investigation. Through systematic ablation studies, we identified SigLIP2-Giant and E5-Small-v2 as the optimal vision and text backbones. We further evaluated fusion strategies ranging from simple concatenation to adaptive gating to determine the best method for integrating these modalities. Our proposed approach utilizes a gated fusion mechanism and achieved a Top-1 accuracy of 84.28\% and an Equal Error Rate of 0.0422 on a comprehensive test protocol. These results represent an 11\% improvement over leading unimodal baselines and demonstrate that integrating synthesized semantic descriptions significantly refines decision boundaries in large-scale pet re-identification.

Problem

• animal identification
• multimodal learning
• visual recognition
• semantic priors
• pet re-identification

Innovation

• multimodal fusion
• synthetic textual descriptions
• gated fusion mechanism
• animal re-identification
• systematic ablation study