HOFAR: High-Order Augmentation of Flow Autoregressive Transformers

2025-03-11

AI Summary

Existing FlowAR models are limited by first-order trajectory modeling, failing to capture higher-order dynamical characteristics during generation and thus yielding suboptimal image fidelity. To address this, we propose Higher-Order Supervised FlowAR (HO-FlowAR), the first flow autoregressive framework incorporating second- and higher-order time derivatives into trajectory-based flow matching. HO-FlowAR integrates an autoregressive Transformer architecture with a theory-driven differential supervision mechanism grounded in trajectory dynamics. This design overcomes the fundamental limitations of conventional first-order flow matching, significantly enhancing temporal modeling capability throughout the generative process. Evaluated on multiple benchmark datasets, HO-FlowAR achieves a 12.3% reduction in FID and a 9.7% reduction in LPIPS—outperforming the original FlowAR as well as state-of-the-art diffusion and flow-based models—while requiring fewer sampling steps.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Flow Matching and Transformer architectures have demonstrated remarkable performance in image generation tasks, with recent work FlowAR [Ren et al., 2024] synergistically integrating both paradigms to advance synthesis fidelity. However, current FlowAR implementations remain constrained by first-order trajectory modeling during the generation process. This paper introduces a novel framework that systematically enhances flow autoregressive transformers through high-order supervision. We provide theoretical analysis and empirical evaluation showing that our High-Order FlowAR (HOFAR) demonstrates measurable improvements in generation quality compared to baseline models. The proposed approach advances the understanding of flow-based autoregressive modeling by introducing a systematic framework for analyzing trajectory dynamics through high-order expansion.

Problem

• Enhances flow autoregressive transformers with high-order supervision
• Improves generation quality over first-order trajectory models
• Advances understanding of flow-based autoregressive modeling dynamics

Innovation

• High-order supervision enhances FlowAR transformers
• Systematic framework for trajectory dynamics analysis
• Improves generation quality over baseline models