Follow-Your-Shape: Shape-Aware Image Editing via Trajectory-Guided Region Control

2025-08-11

AI Summary

Existing flow-based image editing methods often fail on large-scale shape transformation tasks—either failing to achieve the desired deformation or compromising background integrity. This paper introduces the first training-free, mask-free framework for large-scale image shape editing. Methodologically, it models flow trajectories via diffusion models, defines token-level velocity differences to generate Trajectory Difference Maps (TDMs) for precise localization of editable regions, and incorporates a scheduled KV-injection mechanism to enable stable, fine-grained regional control. Evaluated on the newly constructed ReShapeBench benchmark, our approach significantly outperforms state-of-the-art methods, particularly in large-scale structural replacement tasks. It achieves superior editing accuracy and visual fidelity while operating entirely in an unsupervised, mask-free setting—marking the first demonstration of high-fidelity shape editing without supervision or explicit segmentation masks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

While recent flow-based image editing models demonstrate general-purpose capabilities across diverse tasks, they often struggle to specialize in challenging scenarios -- particularly those involving large-scale shape transformations. When performing such structural edits, these methods either fail to achieve the intended shape change or inadvertently alter non-target regions, resulting in degraded background quality. We propose Follow-Your-Shape, a training-free and mask-free framework that supports precise and controllable editing of object shapes while strictly preserving non-target content. Motivated by the divergence between inversion and editing trajectories, we compute a Trajectory Divergence Map (TDM) by comparing token-wise velocity differences between the inversion and denoising paths. The TDM enables precise localization of editable regions and guides a Scheduled KV Injection mechanism that ensures stable and faithful editing. To facilitate a rigorous evaluation, we introduce ReShapeBench, a new benchmark comprising 120 new images and enriched prompt pairs specifically curated for shape-aware editing. Experiments demonstrate that our method achieves superior editability and visual fidelity, particularly in tasks requiring large-scale shape replacement.

Problem

• Achieving large-scale shape transformations in image editing
• Preventing unintended alterations to non-target regions during edits
• Ensuring precise and controllable object shape modifications

Innovation

• Training-free mask-free shape editing framework
• Trajectory Divergence Map for editable regions
• Scheduled KV Injection for stable editing