representation learning

This work investigates whether large language models (LLMs) can generalize in-context learning (ICL) capabilities—traditionally confined to discrete text—to continuous vector inputs. To this end, we propose Vector-ICL, a novel paradigm that aligns continuous vectors from multimodal black-box encoders into the LLM’s embedding space via lightweight, learnable projectors, enabling tuning-free, cross-modal vector-level ICL. Our key contribution is demonstrating that standard, language-modeling-pretrained LLMs—combined with modality-agnostic projectors—can achieve zero-shot or few-shot generalization to unseen continuous vectors, bypassing tokenization constraints entirely. Evaluated across eight diverse tasks—including text reconstruction, numerical regression, molecular property prediction, and fMRI decoding—Vector-ICL consistently outperforms conventional few-shot ICL baselines and domain-specific models, substantiating the feasibility and promise of LLMs as universal vector processors.

The evolution of embedding techniques from word vectors to multimodal representations remains fragmented, lacking a unified framework that integrates advances across linguistic, cross-lingual, personalized, and multimodal domains—particularly for embodied multimodal learning in large language models. Method: We systematically survey static and contextual language representations, cross-lingual and personalized modeling, sentence/document embeddings, and multimodal fusion in vision, robotics, and cognitive science. We synthesize recent progress in interpretability, model compression, numerical encoding, and bias mitigation, and propose a novel paradigm emphasizing strong alignment across non-textual modalities and scalable training. Contributions: We construct a comprehensive knowledge graph of end-to-end embedding technologies—from Word2Vec and BERT to GPT, generative topic models, and multimodal alignment/distillation methods—identifying key technical bottlenecks and ethical challenges. This work delivers the first systematic roadmap for multimodal, embodied learning in foundation models.

This work investigates how vision-language models (VLMs) construct modality-agnostic task representations. We introduce the concept of *task vectors*—one-dimensional linear embeddings that compactly encode semantically equivalent task specifications across modalities (images/text) and formats (instructions/examples), outperforming full prompt inputs in downstream performance. Methodologically, we leverage autoregressive VLMs and employ cross-modal transfer evaluation, task vector disentanglement, and linear probing analysis. Key contributions: (1) We provide the first empirical evidence that task vectors exhibit strong generalization—enabling zero-shot transfer from LLMs to VLMs; (2) they can be extracted solely from instruction-based prompts without demonstrations; (3) they support cross-modal task triggering (e.g., image-to-text or text-to-image); and (4) they significantly improve zero-shot generalization across diverse tasks and VLM architectures. Our findings suggest that task semantics in VLMs are linearly separable and highly portable, offering a unified representation for multimodal task specification.

Large language models (LLMs) struggle to represent unipolar categorical concepts (e.g., “is an animal”) due to the absence of natural antonyms and inherent nonlinearity in their embeddings. Method: We propose the Extended Linear Representation Hypothesis—modeling unipolar categories as direction vectors and their semantic scope as convex polyhedra—and establish the first rigorous mathematical mapping between conceptual hierarchy depth and representation geometry (i.e., decreasing inter-vector angles and polyhedral containment). Using WordNet, we construct a hierarchical concept set of 900+ categories and estimate direction vectors in Gemma and LLaMA-3, followed by joint convex-geometric and semantic analysis. Results: Empirically, deeper-category vectors exhibit significantly smaller pairwise angles; moreover, parent-category polyhedra approximately contain those of their children. These findings confirm that LLMs internally organize semantic categories via interpretable geometric structures—offering a novel paradigm for interpretable and hierarchically grounded semantic representation in LLMs.

This work addresses the insufficient robustness of concept representations in large language models (LLMs). We propose the Gaussian Concept Subspace (GCS) framework, which abandons the conventional linear probe assumption that concepts are represented by single vectors, instead modeling concepts as structured subspaces within the model’s latent space. GCS represents each concept as a joint Gaussian distribution parameterized by its mean and covariance matrix, enabling principled subspace confidence estimation and representation-level interventions—e.g., emotion steering. This yields improved semantic stability and conceptual completeness. Extensive experiments across multiple LLM scales and architectures demonstrate that GCS achieves higher faithfulness and plausibility than baseline methods. In emotion-guided generation tasks, GCS simultaneously enhances control precision and text fluency, enabling more controllable and robust concept-level interventions.

This work addresses the limitations of existing representation alignment methods, which predominantly rely on geometric properties and struggle to capture the global structural organization of model representations. To overcome this, the study introduces topological data analysis into the field for the first time, proposing a Mapper-based visual analytics framework. By integrating force-directed layout, Bubble Sets, motif querying, and membrane-inspired heuristics, the framework enables a unified analytical pipeline spanning global structure alignment, local region matching, and fine-grained pattern exploration. Case studies on language and multimodal models, complemented by expert evaluations, demonstrate that the approach effectively reveals and compares the topological organization of representations across different models or layers, offering deep structural insights.

This work addresses the limitations of existing Earth observation foundation models, which predominantly rely on raster data and overlook the structured geographic semantics embedded in open vector datasets such as OpenStreetMap, thereby hindering comprehensive understanding of human–environment systems. To overcome this, we propose the first unified spatial representation learning framework that deeply integrates remote sensing imagery and vector data within a shared embedding space, breaking away from conventional modality-isolated paradigms. By leveraging self-supervised learning and multimodal alignment—while explicitly modeling geometric, topological, and semantic relationships—our approach enables synergistic raster perception and vector-based reasoning. The method substantially enhances accuracy, semantic interpretability, and explainability on downstream tasks, laying a theoretical and methodological foundation for developing human-centered, semantically rich geospatial foundation models.

This work addresses the lack of theoretical understanding regarding how the geometry of task vectors in Transformers is shaped by training distributions and underpins both in-distribution and out-of-distribution reasoning. By training small-scale Transformers from scratch in a controlled synthetic setting, the study integrates task vector analysis with representational geometric modeling to uncover, for the first time, the intrinsic relationship among task vector geometry, training distribution, and generalization behavior. The findings reveal that in-distribution tasks are solved through Bayesian-like retrieval via convex combinations of task vectors, whereas out-of-distribution tasks rely on extrapolation within approximately orthogonal subspaces. Crucially, both mechanisms coexist within the same architecture, establishing a formal theoretical link between the geometric structure of task vectors and the model’s generalization capabilities.

This study investigates whether singular vectors in attention mechanisms can reflect the internal feature representations of language models, addressing a gap in theoretical grounding within interpretability research. By integrating singular value decomposition, theoretical modeling, and sparse attention decomposition, the work provides the first theoretical justification for the alignment between singular vectors and model features, and derives operational criteria testable in real-world models. Empirical analysis in controlled-feature models confirms this alignment, while theoretical analysis characterizes the general conditions under which it holds. Furthermore, the study identifies sparse attention structures in actual language models that conform to these theoretical predictions, offering a novel perspective and a falsifiable explanatory framework for understanding attention mechanisms.