Automated Self-Refinement and Self-Correction for LLM-based Product Attribute Value Extraction

2025-01-02

AI Summary

This study investigates the impact of self-refinement techniques—error-driven prompt rewriting and self-correction—on large language models’ (LLMs) performance in extracting structured attributes from unstructured e-commerce product descriptions provided by suppliers. Method: We systematically evaluate two self-refinement approaches under zero-shot, few-shot, and supervised fine-tuning paradigms, measuring marginal gains in F1 score against computational cost (token consumption). Contribution/Results: Empirical results show that both self-refinement methods yield negligible improvements (<0.8% F1 gain) while substantially increasing token overhead. When sufficient labeled data is available, supervised fine-tuning remains superior. To our knowledge, this is the first work to empirically expose the “high-cost–low-gain” trade-off of self-refinement in attribute extraction, challenging its general efficacy. We further demonstrate that scaling supervised fine-tuning amortizes initial setup costs, establishing it as a pragmatic, cost-effective pathway for structured product information extraction.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Information Extraction
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Structured product data, in the form of attribute-value pairs, is essential for e-commerce platforms to support features such as faceted product search and attribute-based product comparison. However, vendors often provide unstructured product descriptions, making attribute value extraction necessary to ensure data consistency and usability. Large language models (LLMs) have demonstrated their potential for product attribute value extraction in few-shot scenarios. Recent research has shown that self-refinement techniques can improve the performance of LLMs on tasks such as code generation and text-to-SQL translation. For other tasks, the application of these techniques has resulted in increased costs due to processing additional tokens, without achieving any improvement in performance. This paper investigates applying two self-refinement techniques, error-based prompt rewriting and self-correction, to the product attribute value extraction task. The self-refinement techniques are evaluated across zero-shot, few-shot in-context learning, and fine-tuning scenarios using GPT-4o. The experiments show that both self-refinement techniques have only a marginal impact on the model's performance across the different scenarios, while significantly increasing processing costs. For scenarios with training data, fine-tuning yields the highest performance, while the ramp-up costs of fine-tuning are balanced out as the amount of product descriptions increases.

Problem

• Self-refinement Methods
• Large Language Models
• Product Attribute Extraction

Innovation

• Self-refinement methods
• Product attribute extraction
• Large language models