Augmentation techniques for video surveillance in the visible and thermal spectral range

2026-06-11

AI Summary

This work addresses the limited generalization of existing deep models in multispectral (visible and thermal infrared) video surveillance, which stems from sensor discrepancies and the scarcity of thermal imaging data. The authors propose a CNN-based framework for multispectral object detection and systematically design and evaluate several cross-spectral data augmentation strategies—namely thermal feature simulation, texture-preserving transformations, and illumination-invariant enhancement—to effectively integrate color, shape, and thermal radiation cues. Experimental results demonstrate that the proposed approach significantly improves detection accuracy and robustness in mixed-spectral scenarios, validates the auxiliary value of visible-spectrum data for thermal infrared detection, and fills a critical gap in the understanding of cross-spectral data augmentation mechanisms.

Abstract

In intelligent video surveillance, cameras record image sequences during day and night. Commonly, this demands different sensors. To achieve a better performance it is not unusual to combine them. We focus on the case that a long-wave infrared camera records continuously and in addition to this, another camera records in the visible spectral range during daytime and an intelligent algorithm supervises the picked up imagery. More accurate, our task is multispectral CNN-based object detection. At first glance, images originating from the visible spectral range differ between thermal infrared ones in the presence of color and distinct texture information on the one hand and in not containing information about thermal radiation that emits from objects on the other hand. Although color can provide valuable information for classification tasks, effects such as varying illumination and specialties of different sensors still represent significant problems. Anyway, obtaining sufficient and practical thermal infrared datasets for training a deep neural network poses still a challenge. That is the reason why training with the help of data from the visible spectral range could be advantageous, particularly if the data, which has to be evaluated contains both visible and infrared data. However, there is no clear evidence of how strongly variations in thermal radiation, shape, or color information influence classification accuracy. To gain deeper insight into how Convolutional Neural Networks make decisions and what they learn from different sensor input data, we investigate the suitability and robustness of different augmentation techniques...

Problem

• multispectral
• thermal infrared
• object detection
• data augmentation
• convolutional neural networks

Innovation

• multispectral object detection
• thermal infrared imaging
• data augmentation
• convolutional neural networks
• visible-infrared fusion