When One Modality Is Not Enough: Multimodal Sex and Life-Stage Classification of Red Deer from Aerial RGB-Thermal Video

2026-08-07

AI Summary

This study addresses the challenge of accurately classifying red deer by sex and life stage in aerial surveys, where single-modality imagery—either RGB or thermal—is hindered by occlusion, seasonal antler variation, and low resolution. The work presents the first full-stage fusion of RGB and thermal modalities in wildlife aerial census, leveraging self-supervised DINOv3 features for multi-stage modality alignment. It further introduces modality consistency verification and georeferenced body-size calibration, integrated with object tracking and cross-frame voting to jointly infer species, sex, and life stage. Evaluated across four flight campaigns, the method correctly classified 25 out of 26 red deer (96.2%), significantly outperforming single-modality approaches (76.9%) and demonstrating exceptional robustness in sex identification across multiple seasons and complex environmental conditions.

Abstract

Aerial drone surveys increasingly support wildlife population estimation, yet a useful census is more than a count: population dynamics are defined by species composition, sex ratios and age structure, that is, by which species are present and how a herd splits into adult males, adult females and juveniles. We use red deer ($\textit{Cervus elaphus}$) as a test case, because managers act on these dynamics and because the visible cue defining adult males, the antlers, is seasonally variable. Surveys are flown nadir, high enough not to disturb the animals, so each deer occupies only a small, low-resolution patch. The two recording modalities fail in opposite conditions: in color a deer under canopy blends into the ground, while in thermal it becomes a bright blob that loses fine detail. Rather than trust either modality alone, we fuse them at every stage using self-supervised DINOv3 features. Our pipeline tracks animals in both modalities, treats an animal as confirmed only when the two cameras agree, keeps only the clear, non-occluded frames, and assigns species and sex by a vote across them; life stage is read separately from geo-referenced body size, since at survey resolution a juvenile often only differs from an adult female in size. Across four flights spanning the antler season the fused pipeline correctly classifies 25 of the 26 detected individuals (7 of 8 adult males, all 16 adult females and 2 juveniles), against 20 of 26 for either sensor alone. Multimodal species classification reaches 96.0%, while for sex classification fusing the two sensors matters most: the combined RGB+thermal model is the most robust across environments and seasons. Automating the demographic classification turns a drone flight from a count into a repeatable reading of herd structure, so the sex ratios and age structure that managers already act on can be gathered as often as a survey can be flown.

Problem

• multimodal classification
• sex classification
• life-stage classification
• aerial survey
• red deer

Innovation

• multimodal fusion
• self-supervised learning
• aerial wildlife census
• demographic classification
• RGB-thermal video