🤖 AI Summary
This study addresses the limited scalability and reproducibility of existing crop yield prediction methods, which are often confined to specific crops or regions and lack standardized pipelines for multi-source environmental data. To overcome this, we propose UniCrop—the first general-purpose data engineering framework that decouples data specification from implementation—enabling automatic acquisition, cleaning, alignment, and fusion of over 200 environmental variables from sources such as Sentinel-1/2, MODIS, and ERA5-Land through simple configuration. The framework generates analysis-ready features for any location, crop, and time period. Coupled with mRMR feature selection and a LightGBM model, it achieves an R² of 0.6576 (RMSE = 465.1 kg/ha) using only 15 features across 557 rice fields; constrained ensemble learning further improves performance to R² = 0.6604, significantly enhancing model scalability, transparency, and practical utility.
📝 Abstract
Accurate crop yield prediction relies on diverse data streams, including satellite, meteorological, soil, and topographic information. However, despite rapid advances in machine learning, existing approaches remain crop- or region-specific and require data engineering efforts. This limits scalability, reproducibility, and operational deployment. This study introduces UniCrop, a universal and reusable data pipeline designed to automate the acquisition, cleaning, harmonisation, and engineering of multi-source environmental data for crop yield prediction. For any given location, crop type, and temporal window, UniCrop automatically retrieves, harmonises, and engineers over 200 environmental variables (Sentinel-1/2, MODIS, ERA5-Land, NASA POWER, SoilGrids, and SRTM), reducing them to a compact, analysis-ready feature set utilising a structured feature reduction workflow with minimum redundancy maximum relevance (mRMR). To validate, UniCrop was applied to a rice yield dataset comprising 557 field observations. Using only the selected 15 features, four baseline machine learning models (LightGBM, Random Forest, Support Vector Regression, and Elastic Net) were trained. LightGBM achieved the best single-model performance (RMSE = 465.1 kg/ha, $R^2 = 0.6576$), while a constrained ensemble of all baselines further improved accuracy (RMSE = 463.2 kg/ha, $R^2 = 0.6604$). UniCrop contributes a scalable and transparent data-engineering framework that addresses the primary bottleneck in operational crop yield modelling: the preparation of consistent and harmonised multi-source data. By decoupling data specification from implementation and supporting any crop, region, and time frame through simple configuration updates, UniCrop provides a practical foundation for scalable agricultural analytics. The code and implementation documentation are shared in https://github.com/CoDIS-Lab/UniCrop.