image-text alignment

Computational morphology remains underutilized in linguistic fieldwork, revealing a critical disconnect between NLP research and real-world language documentation practice. Method: Guided by user-centered design (UCD), this work systematically diagnoses the roots of this gap and proposes a new research agenda integrating practical utility with scholarly rigor. We enhance the multilingual interlinear glossed text (IGT) generation model GlossLM and conduct a small-scale field user study with language documenters. Contribution/Results: We identify key usability bottlenecks—including insufficient gloss tag standardization, tokenization ambiguity, and lack of personalization support. Results show that while current models achieve strong automatic evaluation scores, they fail to meet documenters’ core needs for interpretability, controllability, and seamless workflow integration. This is the first study to systematically apply UCD principles to computational morphology, establishing a methodological framework and empirical pathway for developing NLP tools purpose-built for language archiving.

Existing visual implicit reasoning methods suffer from unstable intermediate visual evidence due to feature space mismatches. This work proposes the GAP paradigm, which systematically identifies and addresses the norm mismatch between visual latent variables and input embeddings for the first time. Through alignment mechanisms operating at three granularities—feature, context, and capability—the approach enhances the stability and effectiveness of multimodal large language model reasoning. The method incorporates a lightweight PCA-based alignment head for latent variables, verifiable auxiliary visual supervision, and a model-capability-aware selective supervision strategy. Evaluated on Qwen2.5-VL 7B, it achieves state-of-the-art performance in perception-reasoning integration, with intervention experiments confirming that the generated latent variables provide task-relevant visual signals significantly superior to placeholder baselines.

This work proposes a novel metric for evaluating the morphological plausibility of subword segmentations without relying on scarce and inconsistent gold-standard word segmentation data. Instead, it leverages widely available morphosyntactic resources—such as Universal Dependencies and UniMorph—and employs IBM Model 1 to probabilistically align subword units with morphological features. The resulting alignment scores serve as a proxy for morphological well-formedness. The proposed method demonstrates strong correlation with traditional morpheme boundary recall across diverse languages and substantially improves applicability in morphologically complex or low-resource settings, thereby enabling a cross-lingually generalizable framework for morphological evaluation of subword segmentations.

Image–text matching faces a fundamental challenge stemming from the semantic gap between visual (continuous, high-dimensional) and textual (discrete, structured) modalities. To address this, we propose VSD-Align, a novel framework that leverages multimodal large language models (MLLMs) to generate fine-grained Visual Semantic Descriptions (VSDs) as learnable semantic anchors. VSD-Align establishes dual alignment: instance-level feature fusion and prototype-level VSD clustering—enabling cross-modal semantic alignment at the representation level. The method supports end-to-end training without auxiliary annotations and inherently enables zero-shot cross-domain transfer. Extensive experiments demonstrate that VSD-Align significantly outperforms state-of-the-art methods on Flickr30K and MSCOCO. Moreover, it achieves superior generalization in zero-shot cross-domain settings—including news and remote sensing domains—validating its effectiveness and broad applicability.

Existing multimodal RAG systems rely on LLMs to generate textual summaries of images and index only these summaries, leading to loss of critical visual details and contextual information—particularly detrimental to chart-text joint question answering in financial documents. This work proposes a direct multimodal embedding retrieval approach: leveraging models such as CLIP to jointly encode images and text into a shared embedding space, enabling cross-modal vector retrieval without intermediate LLM summarization and its associated information decay. Evaluated on a newly constructed financial report QA benchmark, our method achieves a 13-percentage-point absolute gain in mAP@5 (32% relative improvement) and an 11-percentage-point gain in nDCG@5 (20% relative improvement) over the text-summary baseline, significantly enhancing retrieval relevance and factual consistency of generated answers. Extensive experiments across six mainstream LLMs demonstrate the robustness and generalizability of the proposed approach.

Visual document retrieval faces significant challenges due to dense text, complex layouts, and fine-grained semantic dependencies, which hinder precise information access. This work presents the first systematic survey of the field in the era of multimodal large language models, establishing a comprehensive research framework that encompasses benchmark evaluation, methodological evolution, and future challenges. The study proposes a novel paradigm integrating multimodal embeddings, re-ranking models, retrieval-augmented generation (RAG), and agent-based systems to clarify the technological trajectory, identify critical bottlenecks, and offer a clear roadmap for advancing multimodal document intelligence.

This work addresses the significant modality gap between medical imaging and clinical text in shared representation spaces, which leads to insufficient semantic alignment and hampers cross-modal retrieval and understanding performance. To tackle this challenge, the authors propose a modality-agnostic contrastive learning framework that systematically mitigates modality discrepancies in medical settings through optimized embedding space geometry and joint modeling strategies. This approach overcomes the limitations of conventional CLIP-based methods in medical domains and achieves, for the first time, a general and efficient semantic alignment between medical images and clinical text. Experimental results demonstrate substantial improvements in both cross-modal retrieval accuracy on radiology images paired with clinical reports and the quality of generated image captions.

Existing cross-modal retrieval benchmarks primarily focus on coarse-grained or single-condition alignment, falling short in addressing real-world user queries that involve multiple constraints and fine-grained specifications expressed in natural language. To bridge this gap, this work proposes MCMR—the first benchmark for multi-condition, fine-grained, and composable cross-modal retrieval—spanning five product domains and emphasizing constraint awareness and interpretability. We employ a multimodal large language model (MLLM) as both the retriever and a pointwise re-ranker, integrating visual features with long-form textual metadata for joint verification. Experiments demonstrate that visual cues dominate top-ranked accuracy, textual metadata enhances ranking stability for long-tail items, and MLLM-based re-ranking substantially improves fine-grained matching performance, thereby filling a critical evaluation gap in complex query scenarios.

This study addresses a critical limitation in existing retrieval-augmented generation (RAG) benchmarks: their inability to disentangle performance gains stemming from improvements in retrieval mechanisms versus those arising from enhanced document representations. To isolate the impact of document preprocessing, the authors fix the retriever—using BM25 as a consistent baseline—and systematically evaluate diverse document transcription and preprocessing strategies across multilingual and visually dense RAG tasks. Their experiments reveal that optimizing document representation alone substantially narrows the performance gap between BM25 and state-of-the-art multimodal retrievers, indicating that much of the observed gain in current systems originates from representation quality rather than retrieval algorithmic advances. Based on these findings, the work advocates for a new benchmarking paradigm that decouples document transcription from retrieval capability to enable more precise evaluation of RAG components.

Existing approaches struggle to uncover the semantic content of visual tokens across different layers of large language models (LLMs), hindering a deeper understanding of vision–language representation alignment. This work proposes LatentLens, a method that maps visual tokens into a large-scale contextualized text embedding space and leverages k-nearest neighbor retrieval to generate human-readable natural language descriptions. For the first time, LatentLens enables high-granularity interpretation of visual tokens across all model layers and multiple vision–language models (VLMs). Experiments demonstrate that LatentLens significantly outperforms baseline methods such as LogitLens, revealing that the vast majority of visual tokens maintain clear and fine-grained semantics throughout all layers in ten diverse VLMs, thereby providing strong evidence for the high degree of alignment in multimodal representations.