A Lightweight and Explainable Vision-Language Framework for Crop Disease Visual Question Answering

2026-01-08

arXiv.org

AI Summary

This work proposes a lightweight and interpretable vision-language framework tailored for visual question answering in crop disease diagnosis, addressing challenges in both visual understanding and language generation accuracy as well as model interpretability. The approach integrates a Swin Transformer-based visual encoder with a sequence-to-sequence language decoder, enhanced by task-oriented visual pretraining and a two-stage training strategy to strengthen cross-modal alignment and visual representation learning. Evaluated on a large-scale crop disease dataset, the model achieves superior performance—outperforming existing large-scale baselines with significantly fewer parameters—in both crop and disease identification and natural language answer generation, as measured by BLEU, ROUGE, and BERTScore metrics. Its interpretability and robustness are further validated through Grad-CAM visualizations and token-level attribution analysis.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Visual question answering for crop disease analysis requires accurate visual understanding and reliable language generation. This work presents a lightweight vision-language framework for crop and disease identification from leaf images. The proposed approach combines a Swin Transformer vision encoder with sequence-to-sequence language decoders. A two-stage training strategy is adopted to improve visual representation learning and cross-modal alignment. The model is evaluated on a large-scale crop disease dataset using classification and natural language generation metrics. Experimental results show high accuracy for both crop and disease identification. The framework also achieves strong performance on BLEU, ROUGE and BERTScore. Our proposed models outperform large-scale vision-language baselines while using significantly fewer parameters. Explainability is assessed using Grad-CAM and token-level attribution. Qualitative results demonstrate robust performance under diverse user-driven queries. These findings highlight the effectiveness of task-specific visual pretraining for crop disease visual question answering.

Problem

• crop disease
• visual question answering
• vision-language framework
• explainability
• leaf image

Innovation

• lightweight vision-language model
• two-stage training
• cross-modal alignment
• explainable AI
• crop disease VQA