image captioning

Current vision-language models (VLMs) exhibit strong perceptual capabilities but suffer from inherent limitations—including weak logical reasoning, inability to update knowledge without costly retraining, and poor interpretability. Method: This paper systematically surveys neural-symbolic integration approaches and proposes a fine-tuning-free collaborative reasoning framework: a pre-trained VLM serves as the neural frontend, tightly coupled with symbolic components—such as knowledge graphs and formal rule engines—to enable plug-and-play external knowledge injection and transparent, multi-step reasoning. Contribution/Results: We introduce the first taxonomy of neural-symbolic methods specifically designed for VLM enhancement, precisely delineating the applicability boundaries and bottlenecks of each technique in multimodal understanding. The framework provides a systematic solution to improve model interpretability, dynamic knowledge integration, and structured reasoning performance, advancing the state of explainable and adaptable multimodal AI.

This paper addresses key challenges in multimodal large language models (MLLMs)—namely limited scalability, insufficient robustness, and difficulties in cross-modal alignment—within vision-language tasks. To tackle these, it proposes a comprehensive technical taxonomy covering the full MLLM stack and introduces a tripartite evaluation framework centered on scalability, robustness, and ethical alignment. Methodologically, the work conducts a systematic comparative analysis of core techniques, including Transformer-based fusion architectures, visual encoders (e.g., ViT, CLIP), cross-modal alignment learning, instruction tuning, and multi-stage training. It further synthesizes structured best practices for modeling across text, image, video, and audio modalities. The study delivers 12 representative application cases, empirically validating its framework and guidelines. Collectively, this work advances trustworthy deployment and cross-domain adoption of MLLMs through rigorous architectural analysis, principled evaluation, and actionable implementation insights.

This work addresses the challenge of simultaneously achieving structural precision, semantic interpretability, and identity controllability in existing 3D/4D scene representations. We propose “Scene Language”—a unified 3D/4D scene representation framework that integrates executable program structures, natural-language semantic tokens, and visual identity embeddings. To our knowledge, this is the first method enabling zero-shot cross-modal reasoning: without fine-tuning, it directly synthesizes structured scene programs from pretrained language models and vision encoders, while explicitly modeling hierarchical relationships to support fine-grained editing. The representation is renderer-agnostic, interfacing seamlessly with traditional, neural, and hybrid renderers to produce high-fidelity images. Experiments demonstrate significant improvements over baselines—including scene graphs—on complex scene generation tasks, achieving breakthroughs in fidelity, controllability, and editability.

Vision-language models (VLMs) often lack logical consistency in systematic visual reasoning. To address this, we propose Vision-Language Programs (VLP), a framework that automatically compiles VLM-generated natural language descriptions into executable neural-symbolic programs for structured perception and formal reasoning over images. VLP integrates program synthesis, neural-symbolic computation, and structured prompting to balance perceptual flexibility with logical rigor, enabling human-interpretable reasoning traces and mitigating shortcut learning. Evaluated on both synthetic and real-world benchmarks, VLP significantly outperforms direct prompting and structured prompting baselines, improving accuracy and output consistency on complex visual reasoning tasks. Notably, VLP achieves the first end-to-end generation of image-executable programs from natural language instructions, bridging high-level semantics with grounded, verifiable computation.

This work proposes a unified vision model that transcends task-specific architectures traditionally required in computer vision by formulating diverse visual tasks—including detection, segmentation, and geometric prediction—as multimodal generation problems. The model is driven solely by natural language instructions (optionally augmented with visual prompts) to produce text, images, or hybrid outputs from a single architecture, eliminating the need for specialized heads or modular designs. It introduces the SenseNova-Vision Corpus, a large-scale dataset of vision-language instruction-response pairs, and leverages off-the-shelf pretrained multimodal models refined through instruction tuning and joint multimodal generation strategies. This end-to-end framework supports compositional, language-defined tasks and achieves performance on par with or superior to specialized systems across structured understanding, dense geometric prediction, segmentation, and multi-view geometry benchmarks.

This work investigates the fundamental causes underlying the performance limitations of multimodal language models (MLMs) on perception-intensive tasks. Through zero-shot evaluation, feature visualization, and controlled ablation experiments, we systematically analyze the encoding, propagation, and activation mechanisms of visual key-value tokens in state-of-the-art models—including LLaVA-OneVision, Qwen2.5-VL, and Llama-3-LLaVA-NeXT. We identify three key findings: (1) input-agnostic visual key tokens introduced in later layers induce perceptual degradation; (2) text prefixes dynamically modulate visual representations, substantially enhancing perception; and (3) substantial internal visual information remains underutilized—33.3% of critical perceptual signals in the BLINK art-style task fail to activate, and fine-tuned MLMs exhibit weaker visual representations than the original SigLIP encoder. Crucially, we demonstrate for the first time that image value tokens alone suffice for zero-shot segmentation and semantic correspondence, revealing a core bottleneck: “sufficient encoding but insufficient utilization” of visual information.

This work investigates how visual tokens are processed within the language model component of vision-language models (VLMs). Focusing on the LLaVA architecture, we propose a three-tier interpretability framework: (1) Through hierarchical token attribution and representation alignment visualization, we first observe that visual token representations progressively align with the textual vocabulary space across layers; (2) We identify critical semantic information concentrated in the final visual token, revealing a fact-retrieval–like predictive mechanism analogous to pure language models; (3) Systematic ablation experiments demonstrate that removing object-specific visual tokens degrades recognition accuracy by over 70%, while representational interpretability markedly increases with network depth. Our study establishes the first fine-grained, cross-layer, token-level interpretability paradigm for vision–language fusion in multimodal models, offering novel mechanistic insights into how visual information is encoded, aligned, and utilized within language decoders.

This work addresses the challenge of inconsistent semantic representations across multimodal data—such as images, videos, and text—by proposing a language-centric atomic propositional representation framework. The approach transforms observations from any modality into sets of atomic propositions, which are then mapped via a global semantic codebook into a unified, interpretable shared semantic space. This enables compositional expression ranging from fine-grained facts to high-level concepts and facilitates cross-modal reasoning. Experimental results demonstrate that the framework substantially enhances complex multimodal understanding, structured retrieval, and high-quality data curation in autonomous driving and open-world scenarios, offering strong advantages in interpretability, compositionality, and cross-modal alignment.

This work investigates whether images can serve as a self-contained medium for reasoning, independent of textual intermediaries. To this end, it introduces an optical reasoning framework that systematically treats images as complete carriers of logical inference, constructing structured visual reasoning pathways through typographic and graphical representations. This approach enables end-to-end reasoning without relying on intermediate textual steps and unifies the expression of reasoning across both language and multimodal tasks. Evaluated on mathematical, scientific, and multimodal benchmarks, the method matches or surpasses conventional text-based chain-of-thought reasoning. It achieves notable token efficiency, reducing reasoning tokens by 28.57% on average in language tasks and by 16% in multimodal tasks, yielding a 1.96× improvement in token efficiency over standard textual reasoning.

Vision models (e.g., ViT) exhibit significantly weaker reasoning and in-context learning capabilities compared to language models, primarily because conventional patchification lacks semantic structure, hindering global contextual modeling and faithful representation of data distributions. Method: This paper introduces the “visual vocabulary” paradigm, treating image objects as fundamental semantic units and proposing **object-level masked image modeling**—the first semantic grounding pretraining objective operating at the object level. Leveraging MLLM-driven multi-granularity supervision, it explicitly models inter-object semantic relationships and contextual dependencies. Contribution/Results: The approach achieves substantial performance gains on visual reasoning benchmarks—including VQA, GQA, and ScienceQA—demonstrating that object-level semantic modeling effectively enhances global understanding and reasoning capacity. It establishes a novel paradigm for building semantic-aware visual encoders, bridging a critical gap between vision and language modeling principles.

This study addresses the underperformance of vision-language models on fine-grained visual tasks, attributing it to their overreliance on semantic anchors that map easily into linguistic space while neglecting hard-to-name visual details. Through evaluation on visual correspondence tasks and Logit Lens analysis, the authors identify this limitation as stemming from language-based shortcuts learned during training. To mitigate this issue, they propose two strategies: assigning arbitrary labels to non-namable regions to activate richer visual representations, and task-specific fine-tuning to decouple linguistic priors from visual reasoning. Experiments demonstrate that model performance is significantly stronger on namable entities than on non-namable ones; arbitrary labeling effectively improves accuracy, and fine-tuning further enhances generalization, thereby reducing dependence on language priors.

This work addresses the challenge of jointly modeling multimodal understanding and generation within a unified visual framework, bridging the representation gap between text and images while enabling bidirectional cross-modal translation. We propose the first fully vision-native unified diffusion model: both text and images are encoded as RGB pixel sequences, establishing pixel-level bidirectional mapping; a rectified flow-driven diffusion Transformer architecture is adopted, with rendered text images serving as visual conditioning, and lightweight task embeddings enable multi-task joint training. Our approach achieves end-to-end visual unification at the model, task, and representation levels. It significantly improves semantic alignment in text-to-image and image-to-text generation, enhances visual understanding accuracy, and exhibits emergent fine-grained controllability and cycle-consistent cross-modal translation capability.