adaptive visual-textual alignment

This work challenges the prevailing view that image and text representations in vision-language models align only in deep layers. Inspired by DeepDream, the authors propose a synthesis method within an adapter-based architecture that extracts textual concept vectors layer by layer and optimizes corresponding images without auxiliary models or additional datasets. For the first time, this approach provides direct, concept-level evidence of cross-modal alignment starting from the very first layer: across seven network layers and hundreds of concepts, over 50% of images synthesized from the first layer already exhibit clear, salient visual features of target concepts—such as animals, activities, and seasons. This method offers an efficient and novel pathway toward enhancing the interpretability of vision-language models.

Current text-to-image generative models exhibit fundamental deficiencies in compositional logical reasoning, specifically failing to correctly synthesize negation, counting, and spatial relations—despite handling each individually. Method: We conduct a systematic empirical evaluation and architectural analysis to diagnose the root causes of this compositional failure. Contribution/Results: We identify three primary sources: (1) scarcity of compositional patterns in training data; (2) inherent limitations of continuous attention mechanisms in modeling discrete logical operations; and (3) evaluation metrics biased toward visual plausibility rather than logical fidelity. Crucially, we demonstrate that standard interventions—such as data augmentation or fine-tuning—fail to bridge this “compositional gap.” Genuine progress requires rethinking representational formalisms and reasoning mechanisms, not incremental architectural refinements. Our findings provide theoretical insights and methodological guidance for developing multimodal generative models with robust compositional generalization.

Current text-to-image generation models struggle to achieve smooth transitions between semantically similar prompts due to substantial differences in token sequences—particularly in wording, ordering, and conceptual positioning—which hinders effective image blending and continuous editing. This work proposes a Token-to-Token Alignment framework that, without modifying the underlying model, employs a two-stage strategy: first aligning the semantic structures of prompts and then aligning their token embedding representations. By reconstructing diverse prompts into a shared structural form, the method reveals that the latent continuous semantic structure within the text embedding space can be effectively leveraged through representation alignment. Consequently, linear interpolation in this aligned space yields coherent semantic transitions, significantly enhancing the quality of image semantic mixing and continuous editing.

This study investigates the emergence mechanism of “text readability”—the capability to recognize textual content within images—during visual-language model (VLM) training, challenging the implicit assumption that multimodal capabilities evolve synchronously. Method: Leveraging a contrastive learning framework, we systematically analyze multi-stage capability evolution on standard VLM architectures, evaluating performance across diverse vision-language tasks—including those involving rendered text (e.g., screenshots, posters). Contribution/Results: We discover that text recognition capability emerges abruptly in the mid-to-late training phase, whereas semantic understanding develops gradually from early stages. Tasks requiring alignment of images containing rendered text exhibit the slowest convergence. This is the first empirical demonstration of a staged developmental dissociation between symbolic processing (text detection/reading) and semantic processing in multimodal representations. The findings provide theoretical grounding and empirical evidence for optimizing training strategies and enhancing robustness in text-aware visual understanding.

This work addresses fine-grained visual concept composition guided by multiple source images. To overcome limitations of single-image constraints and high fine-tuning costs, we propose a training-free, text-guided multi-image semantic stitching method. Our approach constructs concept-specific subspace projections based on CLIP, mapping linguistically specified semantic regions from multiple reference images into a unified embedding space; these embeddings are then seamlessly stitched and fused via the IP-Adapter framework. This is the first method enabling training-free, cross-domain controllable disentanglement and recomposition of multi-image concepts. Experiments demonstrate substantial improvements over existing image-conditioned generation methods in both conceptual coverage breadth and spatial localization accuracy. Comprehensive qualitative and quantitative evaluations validate the effectiveness and robustness of our approach.

This work addresses the limitations of prevailing vision-language models that employ uniform alignment strategies, which overlook the dynamic disparities in information density and semantic scope between images and text, often resulting in the loss of fine-grained semantics and excessive computational overhead. To overcome these issues, the authors propose a dynamic non-uniform cross-modal alignment framework that leverages a learnable matching function to adaptively assign image regions to textual segments based on their information density. The framework further integrates progressive high-resolution visual feature enhancement with an efficient attention mechanism to achieve precise yet computationally frugal alignment. Extensive experiments demonstrate that the proposed method significantly improves performance on multiple downstream tasks while reducing computational costs, effectively balancing accuracy and efficiency.

This work addresses the challenges of semantic distortion and structural inconsistency in generating images from complex textual prompts involving multiple objects with specified attributes, quantities, and spatial relationships. The authors propose a scene graph–based, zero-shot soft visual guidance mechanism that leverages a lightweight language model during inference to produce conditional signals that steer a diffusion model. The key innovation lies in the ASQL Conditioner module, which enables the first unified zero-shot conditioning framework jointly modeling Attribute, Size, Quantity, and Location. This approach significantly enhances semantic fidelity and structural coherence in generated images under complex prompts while preserving output diversity and computational efficiency.

Text-to-image (T2I) models still struggle with compositional semantic alignment—particularly in object-attribute binding, spatial relations, numeral comprehension, and multi-object generation. To address this, we propose the first unified evaluation framework, systematically benchmarking six representative models across fine-grained dimensions using T2I-CompBench++ and GenEval. Our analysis reveals that Vector-Quantized Autoregressive (VAR) architectures—specifically the Infinity series—significantly outperform dominant diffusion-based models in compositional generation. Notably, the parameter-efficient Infinity-2B already surpasses SDXL and PixArt-α, while Infinity-8B achieves state-of-the-art overall performance. This work establishes the first cross-architectural, fair comparison between VAR and diffusion models, empirically validating that structural priors—encoded via autoregressive token modeling—yield simultaneous gains in both generation fidelity and computational efficiency.

This work addresses the slow convergence and degraded generation quality in standard Flow Matching training caused by gradient conflicts arising from data heterogeneity. It is the first to model the Flow Matching objective as a dynamic quadratic form dominated by the Neural Tangent Kernel (NTK), thereby revealing how heterogeneous data interact within the residual vector field. Building on this insight, the authors propose a semantic granularity alignment strategy that explicitly modulates feature cross-terms to mitigate gradient interference. The method significantly accelerates convergence on both DiT and U-Net architectures while enhancing the structural integrity of generated images, achieving a superior trade-off between training efficiency and sample quality.

Text-to-image (T2I) models excel in visual fidelity but suffer from poor compositional generalization—particularly in modeling object relations, attribute binding, and fine-grained details—largely due to the absence of explicit discriminative training on compositionally similar prompts or images. To address this, we propose an embodied multi-tool LLM agent framework that autonomously constructs high-discriminability compositional contrastive datasets. We further introduce Agentic Preference Optimization (APO), an agent-driven preference tuning method that jointly leverages image generation, editing, and visual question answering tools, integrated with reward modeling for end-to-end compositional reasoning—without degrading visual quality. Evaluated on T2I-CompBench and related benchmarks, our approach achieves state-of-the-art performance. Notably, it also yields unexpected improvements in auxiliary capabilities such as text rendering, despite no explicit optimization for these tasks.