Forecasting the Number of Harvest-ready Fruits of Sweet Peppers Using Multimodal Time-Series Data

2026-07-22

AI Summary

This study addresses the scarcity of publicly available datasets that jointly capture visual growth dynamics and per-plant fruit count annotations for bell pepper yield prediction. To bridge this gap, the authors introduce the first multimodal time-series dataset comprising 691 bell pepper plants across two growing seasons, annotated with fruit maturity stages. They propose a deep learning framework that integrates DinoV2-derived visual features with numerical counts using an LSTM architecture, enabling harvestable fruit prediction at irregular sampling intervals. The model incorporates deep ensembles and Gaussian negative log-likelihood uncertainty calibration. Compared to a persistence baseline, it reduces RMSE by 33% and 38% in the 2022 and 2023 seasons, respectively, achieving uncertainty calibration errors (UCE) of 0.39–0.89, thereby significantly enhancing both predictive accuracy and decision reliability.

Abstract

Accurate yield forecasting at the individual-plant level is critical for precision agriculture and supply-chain planning, yet public datasets capturing both visual growth dynamics and per-plant measurement labels are scarce. In this paper, we introduce a novel, annotated image time-series dataset of 691 sweet pepper plants monitored over two growing seasons, comprising 4837 images with per-plant fruit counts categorized by maturity. We propose a multimodal deep learning framework that fuses high-dimensional image features, extracted using the DinoV3 encoder, with numerical count measurements. Our architecture utilizes a Long Short-Term Memory (LSTM) network to model temporal dependencies and handles irregular sampling intervals common in greenhouse monitoring. Through quantitative experiments, we demonstrate that this multimodal approach reduces RMSE over a persistence baseline by 33% and 38% in the 2022 and 2023 seasons, respectively, with a further 1.2% average gain over a measurement-only model. Furthermore, we employ Deep Ensembles and Gaussian Negative Log-Likelihood (NLL) to provide calibrated uncertainty estimates, with an Uncertainty Calibration Error (UCE) ranging from 0.39 to 0.89 depending on the cross-season evaluation direction, offering a principled confidence signal for real-world agricultural decision-making. We release the dataset and code to support reproducible research and to accelerate development of data-driven yield forecasting methods for horticultural crops.

Problem

• yield forecasting
• precision agriculture
• multimodal time-series
• sweet pepper
• fruit counting

Innovation

• multimodal time-series
• DinoV3 encoder
• LSTM
• Deep Ensembles
• uncertainty calibration