Food Image Segmentation with LLM-Derived Ingredient Labels and Multimodal Fusion

2026-07-28

AI Summary

Existing food image segmentation methods struggle with visually similar or rare ingredients. This work proposes a multimodal segmentation framework that does not require pre-aligned image–text pairs, uniquely integrating semantic labels generated by a large language model (LLM) directly into both the feature representations and query embeddings of the segmentation model. The approach introduces two plug-and-play modules—LIM-F and LIM-Q—that are compatible with both CNN and Transformer backbones. Evaluated on FoodSeg103, the method achieves state-of-the-art performance with an mIoU of 55.0. Notably, LIM-F alone improves the CNN baseline from 47.7 to 49.8 mIoU, with additional GPU memory consumption not exceeding 3.8 GB.

Abstract

Food image segmentation plays a vital role in health-related applications such as nutrition tracking and personalized health monitoring. However, existing models often underperform on visually similar ingredients and rare food categories. To address this issue, we propose two plug-and-play multimodal modules that enhance the segmentation performance by leveraging ingredient labels inferred from food images using large language models (LLMs). The first module, called LIM-F (Language Injection Module for Features), is designed to pair with any image encoder that produces multi-layer outputs (e.g., Swin Transformer), while the second module, LIM-Q (Language Injection Module for Queries), targets Mask2Former-style Transformer-based decoders. Both modules enable training without the need for pre-aligning images with text by directly injecting semantic ingredient information into the visual analysis pipeline. On the FoodSeg103 benchmark, the proposed method achieves state-of-the-art performance. Specifically, integrating LIM-Q into the Mask2Former decoder with a Swin-L image encoder yields a mean Intersection over Union (mIoU) of 55.0. LIM-F also demonstrates strong generalization and competitive performance, reaching an mIoU of 54.4 under the same model (Swin-L+Mask2Former). Furthermore, its applicability extends beyond Transformer-based decoders, as evidenced by an improvement from 47.7 to 49.8 mIoU when integrated into a CNN-based architecture. Notably, the improved segmentation accuracy is achieved with only a moderate (at most 3.8 GB) increase in the GPU memory consumption during training. Thus, the proposed approach offers a practical and scalable solution for fine-grained food understanding.

Problem

• food image segmentation
• visually similar ingredients
• rare food categories
• fine-grained food understanding

Innovation

• food image segmentation
• large language models
• multimodal fusion
• language injection module
• fine-grained food understanding