ReGraph: Learning to Generate Recipe Graphs from Food Images

2026-08-07

AI Summary

Existing recipe generation methods rely on free-form text, making it difficult to explicitly model ingredient state transitions, procedural sequencing, and inter-entity dependencies, and thus lack interpretable, structured procedural knowledge. This work proposes Recipe Graph Learning, a framework that constructs ReGraph—a large-scale recipe graph dataset—and introduces, for the first time, Recipe Reasoning Chains (RR-CoT) as supervision signals alongside pattern-based structured graph representations to transform implicit cooking procedures into evaluable entity-relation graphs. Through a two-stage training strategy that integrates multimodal large language models with deterministic pattern matching, the approach significantly enhances the generation quality of cooking entities and process relations on mainstream LMM backbones. The results demonstrate that structured outputs outperform purely textual generation, while also highlighting fine-grained ingredient state modeling as a critical remaining challenge.

Abstract

Recent Large Multimodal Models (LMMs) have achieved impressive performance in recipe generation from food images.However, cooking is a structured transformation process in which ingredients undergo state changes through ordered actions,while free-form recipe language leaves the corresponding entities, intermediate states, and dependencies largely implicit and entangled.A graph representation makes this procedural knowledge explicit and compositional, providing a structured basis for assessing whether model outputs encode process-level knowledge rather than merely presenting plausible textual descriptions. To address this limitation, we present ReGraph, a large-scale recipe graph dataset that represents ingredients, cooking actions, and tools as entities, uses entity attributes to describe ingredient state changes, and employs typed relations to encode manipulation targets, destinations, and procedural ordering. ReGraph further incorporates explicit Recipe Reasoning Chain-of-Thought (RR-CoT) traces, providing auxiliary supervision for procedural decomposition and structured graph generation. Building on ReGraph, we propose Recipe Graph Learning (RGL), a two-stage framework that enables LMMs to generate a plausible fine-grained cooking workflow from a food image in the form of a structured recipe graph. Under a deterministic, schema-aware matching protocol, our experiments reveal a substantial gap between text-generation quality and recoverable procedural structure: recipes produced by existing approaches achieve competitive text-generation scores yet yield limited reference-aligned entity and relation structure under the ReGraph schema. In contrast, across two representative LMM backbones, RGL consistently improves the generation of cooking entities and procedural relations, while our analysis further shows that fine-grained ingredient-state capture remains the most challenging dimension.

Problem

• recipe generation
• procedural knowledge
• structured representation
• food image understanding
• cooking workflow

Innovation

• recipe graph
• procedural knowledge
• structured generation
• multimodal reasoning
• state change modeling