Language-Guided Object Search in Agricultural Environments

2025-03-03

AI Summary

Robot navigation for language-guided object search in agricultural environments faces challenges due to the absence of room-level semantic maps and the complexity of fine-grained object semantics. Method: This paper proposes an end-to-end approach that eliminates reliance on room-level semantic mapping. It leverages large language models (LLMs) to model fine-grained inter-object semantic relationships, constructs a lightweight semantic relation graph, and integrates vision-language cross-modal reasoning for instruction-driven online path planning. Crucially, LLMs are embedded into the closed-loop object search pipeline for the first time in agricultural robotics, enabling navigation decisions based solely on object-level semantic associations. Contribution/Results: Evaluated on the Boston Dynamics Spot platform, the method achieves 84% offline path planning efficiency, 80% task success rate in real-farm deployments, and a weighted path length of 0.67—demonstrating significant improvements in search accuracy and mobility efficiency while substantially reducing human intervention.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search
• Intelligent Robots: Localization, Mapping, and Navigation

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Creating robots that can assist in farms and gardens can help reduce the mental and physical workload experienced by farm workers. We tackle the problem of object search in a farm environment, providing a method that allows a robot to semantically reason about the location of an unseen target object among a set of previously seen objects in the environment using a Large Language Model (LLM). We leverage object-to-object semantic relationships to plan a path through the environment that will allow us to accurately and efficiently locate our target object while also reducing the overall distance traveled, without needing high-level room or area-level semantic relationships. During our evaluations, we found that our method outperformed a current state-of-the-art baseline and our ablations. Our offline testing yielded an average path efficiency of 84%, reflecting how closely the predicted path aligns with the ideal path. Upon deploying our system on the Boston Dynamics Spot robot in a real-world farm environment, we found that our system had a success rate of 80%, with a success weighted by path length of 0.67, which demonstrates a reasonable trade-off between task success and path efficiency under real-world conditions. The project website can be viewed at https://adi-balaji.github.io/losae/

Problem

• Develops robot-assisted object search in farms using LLM.
• Improves path efficiency and reduces travel distance.
• Achieves high success rate in real-world farm tests.

Innovation

• Uses Large Language Model for semantic reasoning
• Leverages object-to-object relationships for path planning
• Achieves high path efficiency and success rate