contextual vector-field modeling

This work investigates the representation of concepts in large language models and their dynamic evolution under contextual influence. Concept representations are modeled as point-cloud manifolds, while contextual effects are formalized as vector fields acting on these manifolds, analyzed through the lens of neural population geometry. The study reveals, for the first time, that language models across different scales and architectures not only share static geometric structures of concepts but also exhibit common dynamic transformation patterns organized along semantic hierarchies. Experiments across six model families demonstrate that concept shifts induced by context are semantically structured, enabling cross-model prediction of such shifts significantly beyond random baselines. These findings uncover a deep geometric consistency in the internal representations of language models, suggesting universal principles underlying their semantic processing.

This study investigates whether the geometric structure of concepts in large language models is fixed by pretraining priors or dynamically shaped by context. Through representational similarity analysis, activation interventions, and cross-model comparisons (Gemma, Qwen), the work demonstrates for the first time that contextual instructions can deliberately construct arbitrary conceptual topologies—such as ring or tree structures—and causally dominate generation behavior in large models (e.g., Gemma-31B, Qwen-27B), with effect sizes ranging from 0.6 to 0.9 in similarity metrics. This influence is not merely an epiphenomenon of representation but a controllable driver of output. In contrast, smaller models fail to reliably exhibit this capability, highlighting a qualitative divergence in how model scale mediates contextual control over conceptual geometry.

How do neural systems perform nonlinear classification of neural manifolds using contextual information? Existing linear readout methods fail to capture context-dependent manifold reconfiguration mechanisms. Method: We derive, for the first time, an exact analytical formula for context-dependent manifold capacity, explicitly coupling manifold geometry with contextual relevance; integrating differential geometry, random matrix theory, and information geometry, we conduct cross-scale validation via synthetic data simulations and electrophysiological recordings from primate V1 and IT cortex. Contribution: We overcome the limitations of linear readouts by establishing an interpretable, nonlinear, context-sensitive theoretical framework for neural computation. For the first time, we quantitatively identify and model context-driven representational reformatting—previously undetectable by conventional methods—in the early layers of deep neural networks. This significantly advances our understanding of distributed neural coding mechanisms, revealing how contextual signals dynamically reshape population-level neural representations.

Traditional calculus and Riemannian geometry struggle to handle non-manifold, noisy real-world data. This work proposes a data-driven framework grounded in diffusion processes that, for the first time, systematically realizes computable formulations of vector calculus and key geometric objects—such as geodesic distances, curvatures, vector field flows, and solutions to partial differential equations—within the paradigm of diffusion geometry. The framework further integrates topological tools from de Rham cohomology and Morse theory. Leveraging efficient numerical linear algebra techniques, it achieves substantial improvements in computational accuracy, noise robustness, and scalability, demonstrating exceptional numerical stability, low computational complexity, and strong robustness across a range of geometric and topological tasks.

Existing mathematical modeling lacks a rigorous, unambiguous ontological foundation, hindering a unified characterization of the mapping between models and real-world phenomena. This paper introduces, for the first time, an axiomatic definition of mathematical models grounded in Hilbert-space operator theory: a model is formalized as a computable operator acting on random variables, systematically unifying theoretical derivation, experimental implementation, and statistical identification. We further establish a geometric correspondence between the model manifold and the prediction surface, exposing intrinsic structural properties and the fundamental nature of model computability. This framework fills a critical gap in the formal ontology of modeling, providing a unified mathematical foundation for interdisciplinary model construction. It significantly enhances the logical rigor of theoretical inference and the reliability of empirical validation.

This work addresses how to uncover interpretable concept manifolds embedded within the stacked representations of language models. The authors propose Manifold Probe, a method that generalizes traditional linear probing to manifold probing by integrating supervised manifold learning with linear predictability analysis. This approach identifies continuous geometric structures in representation space corresponding to high-level concepts—such as time or space—and determines their encoding directions. Beyond merely detecting the presence of such concepts, the method enables causal intervention: manipulating activations along discovered manifold directions directly alters model behavior. Experiments on Llama 2-7B demonstrate that perturbing representations along the extracted temporal manifold significantly shifts the model’s generated outputs regarding the release years of cultural works, thereby validating both the interpretability and causal efficacy of the recovered manifolds.

This work proposes a Riemannian geometry–based semantic modeling approach to structurally represent scientific literature and uncover its knowledge associations and evolutionary trajectories. Documents are embedded into a two-dimensional knowledge manifold using character-level n-gram TF-IDF vectors, and knowledge positioning, directional gradient analysis, and synthetic research directions are achieved through constrained stress minimization, smoothed particle hydrodynamics (SPH) interpolation, Gaussian process regression, and geodesic optimization. The method innovatively integrates Riemannian geometry with SPH interpolation for semantic modeling of scholarly texts, enabling cross-domain conceptual bridging and the generation of semantically coherent synthetic papers. Evaluated on datasets from composite materials and aerospace engineering, the approach successfully reproduces established research clusters and yields geodesic paths that exhibit plausible conceptual transitions, thereby demonstrating its effectiveness and novelty.

This work addresses the lack of a rigorous definition for coherent structures in large-scale streamline data, which hinders efficient interactive exploration. The authors propose a web-based interactive system whose core innovation lies in constructing a Curve Segment Neighborhood Graph (CSNG) to encode adjacency relationships among streamline segments. By integrating rapid community detection with an enhanced force-directed layout, the system enables multi-scale structural discovery. Leveraging adjacency matrix compression and parallel processing techniques, it achieves real-time, in-browser interactivity on datasets comprising hundreds of thousands of streamline segments, effectively revealing spatial clusters and coherent patterns within complex vector fields.

This work addresses the challenges of modeling complex dependencies and mitigating redundancy in high-dimensional parameter spaces for discrete data generation. It introduces, for the first time, a Riemannian geometric structure with isometric properties into the exponential parameter space of product manifolds over categorical distributions, thereby constructing a low-dimensional latent subspace. By leveraging the Riemannian metric, geodesics within this subspace become straight lines, enabling consistent and efficient flow-matching training. The proposed approach substantially reduces the dimensionality of latent variables while preserving strong representational capacity for discrete data distributions. Experimental results demonstrate that the model achieves accurate and efficient discrete data generation using a significantly lower-dimensional latent space, effectively balancing computational efficiency with modeling performance.

This work addresses the lack of a unified geometric interpretation for Transformers, which hinders understanding of their stability, context limitations, and optimization dynamics. Building upon the geometric axiom that token sequences form a discrete 1-manifold equipped with a measure-theoretic lattice, the paper introduces a continuous stochastic differential geometric framework that unifies core components—RMSNorm, RoPE, and Softmax attention—as integro-differential equations on a semantic fiber bundle. It reveals that attention corresponds to a Schrödinger bridge, while SGD manifests as an Itô diffusion violating detailed balance, and establishes a duality principle governing topological stability. Six geometric predictions—including ε⁻¹/² Lipschitz scaling, Poincaré recurrence suppression on the RoPE torus, and phase transitions at context limits—are validated across five mainstream large language models (124M–8B parameters), achieving R² = 1.000 consistently across optimizers.