How to rewrite the stars: Mapping your orchard over time through constellations of fruits

2026-02-04

AI Summary

This work proposes a novel paradigm for long-term fruit tracking in orchard videos across time by leveraging 3D fruit centroid “constellation” matching. Instead of tracking individual fruits, the method models fruit clusters as sparse 3D point cloud descriptors and establishes cross-temporal, cross-view correspondences through multi-view geometry and 6DoF pose estimation. This constellation-based approach effectively mitigates challenges posed by non-rigid deformations, occlusions, and textureless surfaces, significantly enhancing matching robustness. Furthermore, it enables high-precision orchard mapping, autonomous robot localization, and selective harvesting, offering a new perspective for perception systems in agricultural robotics.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Motion & Tracking
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Following crop growth through the vegetative cycle allows farmers to predict fruit setting and yield in early stages, but it is a laborious and non-scalable task if performed by a human who has to manually measure fruit sizes with a caliper or dendrometers. In recent years, computer vision has been used to automate several tasks in precision agriculture, such as detecting and counting fruits, and estimating their size. However, the fundamental problem of matching the exact same fruits from one video, collected on a given date, to the fruits visible in another video, collected on a later date, which is needed to track fruits'growth through time, remains to be solved. Few attempts were made, but they either assume that the camera always starts from the same known position and that there are sufficiently distinct features to match, or they used other sources of data like GPS. Here we propose a new paradigm to tackle this problem, based on constellations of 3D centroids, and introduce a descriptor for very sparse 3D point clouds that can be used to match fruits across videos. Matching constellations instead of individual fruits is key to deal with non-rigidity, occlusions and challenging imagery with few distinct visual features to track. The results show that the proposed method can be successfully used to match fruits across videos and through time, and also to build an orchard map and later use it to locate the camera pose in 6DoF, thus providing a method for autonomous navigation of robots in the orchard and for selective fruit picking, for example.

Problem

• fruit tracking
• temporal matching
• orchard mapping
• 3D point clouds
• precision agriculture

Innovation

• fruit tracking
• 3D point cloud matching
• constellation descriptor
• precision agriculture
• autonomous navigation