ATMS-KD: Adaptive Temperature and Mixed Sample Knowledge Distillation for a Lightweight Residual CNN in Agricultural Embedded Systems

2025-08-27

AI Summary

To address the challenge of deploying lightweight CNNs on resource-constrained agricultural embedded devices, this paper proposes ATMS-KD: an adaptive temperature scheduling and hybrid sample augmentation–based knowledge distillation framework for efficient knowledge transfer from a MobileNetV3-Large teacher to a compact residual CNN student. Its key innovations include dynamically adjusting the distillation temperature across training stages to enhance optimization stability and incorporating multi-strategy sample augmentation to improve feature fidelity. Evaluated on Damask rose maturity classification, the student model achieves 97.11% test accuracy—outperforming the second-best method by 1.60 percentage points—while maintaining only 72.19 ms inference latency on embedded hardware. Validation accuracy remains consistently ≥96.7%, and knowledge retention exceeds 99%, demonstrating substantial improvements over state-of-the-art distillation approaches in both accuracy and efficiency.

Technology Category

Machine Learning: Transfer, Domain Adaptation, Multi-Task LearningComputer Vision: Learning & Optimization for CVSearch and Optimization: Learning to Search

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSystems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applicationsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This study proposes ATMS-KD (Adaptive Temperature and Mixed-Sample Knowledge Distillation), a novel framework for developing lightweight CNN models suitable for resource-constrained agricultural environments. The framework combines adaptive temperature scheduling with mixed-sample augmentation to transfer knowledge from a MobileNetV3 Large teacher model (5.7,M parameters) to lightweight residual CNN students. Three student configurations were evaluated: Compact (1.3,M parameters), Standard (2.4,M parameters), and Enhanced (3.8,M parameters). The dataset used in this study consists of images of extit{Rosa damascena} (Damask rose) collected from agricultural fields in the Dades Oasis, southeastern Morocco, providing a realistic benchmark for agricultural computer vision applications under diverse environmental conditions. Experimental evaluation on the Damascena rose maturity classification dataset demonstrated significant improvements over direct training methods. All student models achieved validation accuracies exceeding 96.7% with ATMS-KD compared to 95--96% with direct training. The framework outperformed eleven established knowledge distillation methods, achieving 97.11% accuracy with the compact model -- a 1.60 percentage point improvement over the second-best approach while maintaining the lowest inference latency of 72.19,ms. Knowledge retention rates exceeded 99% for all configurations, demonstrating effective knowledge transfer regardless of student model capacity.

Problem

• Develop lightweight CNN models for agricultural embedded systems
• Transfer knowledge from large teacher to small student models
• Improve accuracy and efficiency in agricultural computer vision

Innovation

• Adaptive temperature scheduling for knowledge distillation
• Mixed-sample augmentation to enhance training data
• Lightweight residual CNN models for efficiency