Fine-Grained Customized Fashion Design with Image-into-Prompt benchmark and dataset from LMM

2025-09-11

AI Summary

To address the challenge that end users’ lack of fashion expertise leads to ambiguous text prompts and hinders fine-grained garment customization, this paper proposes a conversational image-to-prompt design framework based on large multimodal models (LMMs). Our method introduces the BUG workflow and establishes FashionEdit—the first dataset explicitly aligned with real-world fashion design processes—enabling end-to-end automation from sketch/reference-image interaction, semantically precise prompt generation, to controllable image editing. By integrating multi-granularity control mechanisms with iterative dialog-based feedback, our approach significantly improves alignment between generated outputs and user intent. Experiments on FashionEdit demonstrate superior performance over baselines across three key dimensions: generation similarity, user satisfaction, and design quality. To foster reproducibility and industrial adoption, we publicly release both the source code and the FashionEdit dataset.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Generative AI evolves the execution of complex workflows in industry, where the large multimodal model empowers fashion design in the garment industry. Current generation AI models magically transform brainstorming into fancy designs easily, but the fine-grained customization still suffers from text uncertainty without professional background knowledge from end-users. Thus, we propose the Better Understanding Generation (BUG) workflow with LMM to automatically create and fine-grain customize the cloth designs from chat with image-into-prompt. Our framework unleashes users' creative potential beyond words and also lowers the barriers of clothing design/editing without further human involvement. To prove the effectiveness of our model, we propose a new FashionEdit dataset that simulates the real-world clothing design workflow, evaluated from generation similarity, user satisfaction, and quality. The code and dataset: https://github.com/detectiveli/FashionEdit.

Problem

• Addresses fine-grained customization in AI fashion design
• Overcomes text uncertainty without professional user knowledge
• Automates clothing design customization using image-to-prompt workflow

Innovation

• LMM-based workflow for fine-grained customization
• Image-into-prompt technique for design generation
• Automated fashion design without human involvement