Text-conditioned Segmentation for Tomato Phenotyping via Procedural Synthetic Data

2026-07-20

AI Summary

This work addresses the performance bottleneck in phenotyping greenhouse tomatoes caused by scarce annotated data by proposing a simulation-to-reality (sim-to-real) text-guided segmentation approach. The method leverages procedurally generated, diverse 3D synthetic tomato greenhouse scenes and, for the first time, integrates them with the text-conditioned segmentation capability of Segment Anything Model 3 (SAM 3). By fine-tuning SAM 3 on this synthetic data, the model achieves organ-level segmentation accuracy. Extensive evaluation on multiple real-world datasets demonstrates that this synthetic-data-driven fine-tuning substantially enhances the model’s generalization and prediction confidence in complex agricultural environments. To foster further research in agricultural vision, the authors publicly release both the synthetic dataset and the fine-tuned model weights.

Abstract

Vision-based automation is an excellent candidate for reducing manual labor in greenhouse crop production and phenotyping. However, progress is constrained by the lack of annotated training data. Recent advances in vision-based foundational models have shown promising results in zero-shot generalization to novel domains, but their performance drops in complex agricultural environments. In this work, we present a sim-to-real framework for tomato plant segmentation that combines synthetic data generation with fine-tuning of a foundation model. We model a commercial cherry tomato greenhouse and use it to generate a large-scale synthetic dataset under diverse viewpoints, lighting conditions, and plant morphology. Subsequently, we fine-tune the Segment Anything Model 3 (SAM 3) on the synthetic dataset, specializing its text-conditioned segmentation behavior for greenhouse crop organs while retaining the general visual prior that makes zero-shot transfer possible. By evaluating our framework on multiple real-world greenhouse datasets, we demonstrate that combining synthetic data with SAM 3 fine-tuning significantly improves segmentation performance and model confidence. To support community benchmarking, we publicly release the procedural model, the generated synthetic dataset, and our fine-tuned SAM 3 weights.

Problem

• tomato phenotyping
• text-conditioned segmentation
• synthetic data
• foundation models
• agricultural vision

Innovation

• procedural synthetic data
• text-conditioned segmentation
• sim-to-real transfer
• foundation model fine-tuning
• tomato phenotyping