Fine-tuning Language Models for Recipe Generation: A Comparative Analysis and Benchmark Study

2025-02-04

AI Summary

This work addresses open-domain recipe generation by systematically fine-tuning lightweight language models—including T5-small, SmolLM-360M/1.7B, and Phi-2—and introducing the first comprehensive evaluation framework tailored to recipe generation. Methodologically, the framework integrates standard NLG metrics (BLEU, ROUGE) with domain-specific dimensions: ingredient consistency, step executability, and allergen substitution feasibility—a novel safety-oriented criterion. Experimental results demonstrate that SmolLM variants achieve the optimal trade-off between performance and parameter count, significantly outperforming Phi-2; notably, model scale exhibits a non-monotonic relationship with recipe practicality. The proposed framework substantially enhances both the utility and safety of generated recipes, empirically validating the viability of compact models for specialized natural language generation tasks.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Large-scale security measurements

Abstract

This research presents an exploration and study of the recipe generation task by fine-tuning various very small language models, with a focus on developing robust evaluation metrics and comparing across different language models the open-ended task of recipe generation. This study presents extensive experiments with multiple model architectures, ranging from T5-small (Raffel et al., 2023) and SmolLM-135M (Allal et al., 2024) to Phi-2 (Research, 2023),implementing both traditional NLP metrics and custom domain-specific evaluation metrics. Our novel evaluation framework incorporates recipe-specific metrics for assessing content quality and introduces an approach to allergen substitution. The results indicate that, while larger models generally perform better on standard metrics, the relationship between model size and recipe quality is more nuanced when considering domain-specific metrics. We find that SmolLM-360M and SmolLM-1.7B demonstrate comparable performance despite their size difference, while Phi-2 shows limitations in recipe generation despite its larger parameter count. Our comprehensive evaluation framework and allergen substitution system provide valuable insights for future work in recipe generation and broader NLG tasks that require domain expertise and safety considerations.

Problem

• Fine-tuning small language models
• Developing robust evaluation metrics
• Comparative analysis of recipe generation

Innovation

• Fine-tuning small language models
• Developing domain-specific evaluation metrics
• Introducing allergen substitution approach