Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer

2024-11-16

arXiv.org

AI Summary

Prior work lacks a systematic analysis of the inference mechanisms underlying Masked Generative Transformers (MGTs). Method: This paper introduces the first systematic “design choice set” for MGT inference and proposes an enhanced inference framework for high-resolution image generation, integrating mask reweighting, hierarchical sampling, and diffusion-model-inspired acceleration. Built upon MaskGIT and Meissonic architectures, it unifies masked image modeling, discrete token prediction, diffusion-prior guidance, and adaptive resampling. Contribution/Results: Evaluated on the HPS v2 benchmark, the Meissonic-1024×1024 model achieves ~70% win-rate improvement. All components are modular, plug-and-play, and yield cumulative gains. The work establishes a reproducible, scalable design paradigm and empirical benchmark for efficient MGT inference.

Technology Category

Computer Vision: Diffusion Models for VisionMachine Learning: Mixture of Experts (MoE)Natural Language Processing: Generation

Application Category

Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applicationsSearch and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

Text-to-image diffusion models (DMs) develop at an unprecedented pace, supported by thorough theoretical exploration and empirical analysis. Unfortunately, the discrepancy between DMs and autoregressive models (ARMs) complicates the path toward achieving the goal of unified vision and language generation. Recently, the masked generative Transformer (MGT) serves as a promising intermediary between DM and ARM by predicting randomly masked image tokens (i.e., masked image modeling), combining the efficiency of DM with the discrete token nature of ARM. However, we find that the comprehensive analyses regarding the inference for MGT are virtually non-existent, and thus we aim to present positive design choices to fill this gap. We propose and redesign a set of enhanced inference techniques tailored for MGT, providing a detailed analysis of their performance. Additionally, we explore several DM-based approaches aimed at accelerating the sampling process on MGT. Extensive experiments and empirical analyses on the recent SOTA MGT, such as MaskGIT and Meissonic lead to concrete and effective design choices, and these design choices can be merged to achieve further performance gains. For instance, in terms of enhanced inference, we achieve winning rates of approximately 70% compared to vanilla sampling on HPS v2 with Meissonic-1024x1024.

Problem

• Bridging DM and ARM discrepancies
• Enhancing MGT inference techniques
• Accelerating MGT sampling process

Innovation

• Masked Generative Transformer enhancement
• Inference techniques for MGT
• DM-based sampling acceleration