inductive bias engineering

This paper addresses the fundamental problem of how internal neural network architecture choices govern training dynamics. We propose an enhanced transformation layer featuring constrained signal paths and adaptive correction mechanisms, grounded in spectral sensitivity analysis and fixed-point theory to derive interpretable, principled structural design guidelines. These guidelines systematically uncover intrinsic connections among gradient flow characteristics, representation regularity, and training stability. Methodologically, we conduct empirical studies using both synthetic and structured tasks to validate the design. Results demonstrate substantial improvements in optimization smoothness, generalization robustness, and depth scalability. Our approach validates a paradigm shift from heuristic performance tuning toward deliberate learning dynamic control, offering novel theoretical foundations and practical design principles for neural architecture engineering.

This work addresses the challenge of classifier-free guidance in diffusion models—without additional training, access to historical weights, or class conditioning. We propose Sliding Window Guidance (SWG), which leverages the primary model itself as an auxiliary model, constraining its receptive field via a sliding window to enable spatial dependency modeling and alignment of error patterns. Our key insight is that the auxiliary model need not be highly accurate; it suffices for it to share error characteristics with the primary model while exhibiting stronger response sensitivity—yielding substantial gains in generation quality. We further introduce an error-driven guidance mechanism and weight regularization for stable control. SWG incurs zero training overhead, requires no architectural modifications or conditional inputs, and matches state-of-the-art guided methods on quantitative metrics (e.g., FID, LPIPS). Moreover, it achieves superior performance in human visual preference evaluations.

Although diffusion autoencoders achieve high-quality image reconstruction, the optimization dynamics governing their latent representations remain poorly understood. This work uncovers, for the first time, a bimodal optimization trajectory in early training stages—characterized by competing reconstruction-dominant and disentanglement-dominant modes—and proposes to steer this trajectory by modulating shortcut connections in the U-Net architecture and scheduling noise-level exposure. To this end, we introduce a novel paradigm that integrates a gated residual U-Net with a curriculum-based noise exposure strategy. Our approach substantially enhances representation quality and reduces sensitivity to random seeds across multiple disentanglement benchmarks, while significantly improving segmentation performance in object-centric learning tasks on both synthetic and real-world datasets.

To address excessive computational overhead when deploying vision models on resource-constrained devices, this paper proposes an efficient Vision Transformer (ViT) architecture design framework. Methodologically: (1) it optimizes the input-output data pathway to enhance representational capacity of lightweight models; (2) it restructures the context window of computationally constrained attention mechanisms to improve local-global modeling efficiency; and (3) it leverages the invertibility and explicit probabilistic modeling properties of normalizing flows to enable high-fidelity, low-overhead knowledge distillation. Experiments demonstrate that the proposed approach achieves comparable or superior accuracy on benchmarks such as ImageNet, while requiring significantly fewer parameters and FLOPs. It also substantially reduces inference latency and memory footprint. The framework establishes a scalable new paradigm for efficient visual understanding at the edge.

This work investigates the emergent computational structures in Transformers performing next-token prediction and their explanatory mechanisms for representational geometric features. Method: We propose a theoretical framework of “architecture-constrained parallel Bayesian belief updating,” unifying optimal prediction principles with mechanistic interpretability. Leveraging hidden Markov model (HMM) construction, probability simplex analysis, attention inverse modeling, and constraint-based refinement of optimal prediction equations, we quantitatively predict attention distributions, OV-circuit vector orientations, and embedding manifold geometry. Contribution/Results: Our framework rigorously derives the geometric structure of attention patterns, OV-circuit vectors, and token embeddings, establishing their formal correspondence to Bayesian inference. On controlled HMM tasks, it successfully reproduces and explains canonical geometric representations—including cyclic dynamics and low-dimensional manifolds—demonstrating both quantitative accuracy and mechanistic interpretability of the theoretical predictions.

This work investigates whether the delayed onset of grokking is determined by the emergence of task-specific representational structure. By introducing a structured supervised contrastive loss in a single-layer Transformer, the study provides the first causal evidence that grokking depends on the formation of correct feature-level structure—not merely label-level alignment. Experiments demonstrate that authentic structural priors accelerate grokking by up to 2.75×; even incorrect but consistent structures permit grokking, whereas random structures entirely suppress generalization. Across 95 trials, the emergence of representational structure consistently precedes and reliably predicts the onset of generalization, underscoring the critical role of structural priors in enabling grokking.

This work addresses the challenge that pretrained bridging models in image translation often fail to effectively leverage prior information. To overcome this limitation without requiring retraining, the authors propose a training-free prior-guidance mechanism that enhances prior utilization by introducing unseen weak priors and contrasting them with observed priors. This approach is integrated with a frequency modulation strategy—Frequency Modulation with Prior Guidance (FMPG)—which adaptively scales features in the frequency domain. The study establishes a novel training-free paradigm for prior-guided image translation, elucidates the underlying mechanisms of prior influence during bridging, and constructs a cascaded CFG-FMPG framework. Experimental results demonstrate consistent improvements in both generation quality and inference efficiency across diverse image translation tasks.

Neural networks often exhibit the "grokking" phenomenon during training, characterized by an initial memorization phase followed by sudden generalization, yet its underlying mechanisms and control strategies remain poorly understood. This work reveals that dimensional collapse consistently precedes grokking and, for the first time, leverages the geometric structure of representations as a control signal. We propose Geometric Dimensionality Regularization (GeomDR), a spectral regularization method that modulates the effective representational dimensionality of hidden layers in both multilayer perceptrons and Transformers. Evaluated on modular arithmetic (addition and division) and permutation composition tasks, GeomDR accelerates the onset of generalization by up to 52×, providing strong empirical evidence that geometric intervention effectively regulates grokking dynamics.

This work addresses a critical gap in topological deep learning: the absence of native benchmark datasets that inherently embody higher-order topological structures. Current research often relies on elevating graph-based data to construct such structures, which limits rigorous model evaluation and hinders progress. The study systematically identifies this benchmark deficiency and advocates for the development of genuinely native datasets rooted in intrinsic higher-order topology. Drawing upon established higher-order modeling paradigms—such as message passing frameworks and sheaf theory—the paper proposes concrete directions and standardization guidelines for dataset construction. These contributions aim to establish a solid foundation for topological deep learning and catalyze the future development of benchmarks in higher-order machine learning.

This work addresses the challenge of automatically interpreting the vast number of features extracted by sparse autoencoders (SAEs) in visual models without manual intervention. The authors propose a causal intervention–based feature steering method: by activating individual SAE features in isolation on an empty (zeroed) input image and prompting a vision–language model to describe what it “sees,” semantic explanations for the corresponding visual concepts are generated end-to-end. To further enhance interpretability, they introduce a Steering-informed Top-k fusion strategy that combines the strengths of causal interventions and representative input examples, significantly improving explanation quality without additional computational cost. The approach is highly scalable, with interpretation fidelity increasing alongside the size of the language model, marking the first fully automated pipeline for high-quality visual feature explanation.