CLEAR: Cue Learning using Evolution for Accurate Recognition Applied to Sustainability Data Extraction

2025-01-30

AI Summary

Existing large language models (LLMs) for sustainability data extraction from architectural images rely heavily on manually engineered domain-specific prompts, resulting in low accuracy and poor generalization in non-expert settings. Method: We propose the first prompt optimization framework integrating LLMs with evolutionary computation, introducing two key innovations: variable-length domain representation and real-valued estimation strategies to enable end-to-end adaptive prompt generation. Contribution/Results: The method significantly enhances LLM consistency and generalization in expert visual understanding tasks. On architectural image recognition, it reduces error rates by two orders of magnitude compared to handcrafted prompts—and even surpasses human experts. Ablation studies confirm the framework’s simplicity and computational efficiency, demonstrating robust performance without architectural modifications to the underlying LLM.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evolutionary Computation

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Model (LLM) image recognition is a powerful tool for extracting data from images, but accuracy depends on providing sufficient cues in the prompt - requiring a domain expert for specialized tasks. We introduce Cue Learning using Evolution for Accurate Recognition (CLEAR), which uses a combination of LLMs and evolutionary computation to generate and optimize cues such that recognition of specialized features in images is improved. It achieves this by auto-generating a novel domain-specific representation and then using it to optimize suitable textual cues with a genetic algorithm. We apply CLEAR to the real-world task of identifying sustainability data from interior and exterior images of buildings. We investigate the effects of using a variable-length representation compared to fixed-length and show how LLM consistency can be improved by refactoring from categorical to real-valued estimates. We show that CLEAR enables higher accuracy compared to expert human recognition and human-authored prompts in every task with error rates improved by up to two orders of magnitude and an ablation study evincing solution concision.

Problem

• Large Language Models
• Image Information Extraction
• Expert Prompts Dependence

Innovation

• CLEAR method
• LLM and evolutionary computing
• automatic optimization for image information extraction