auxiliary head design

This work addresses the poor initial performance of active noise control (ANC) systems under abrupt acoustic environment changes, which heavily depends on parameter initialization. For the first time, model-agnostic meta-learning (MAML) is introduced to this domain, enabling rapid adaptation by jointly optimizing the initial parameters of both the control filter and the secondary path model. The proposed method employs a two-stage inner-loop procedure that simulates system identification and noise cancellation, integrated within an FxLMS framework with online secondary path modeling. Notably, only a small amount of real-world path data is required during pre-training. Experimental results demonstrate that, compared to a non-reinitialized baseline, the approach significantly reduces early-stage error, shortens convergence time, lowers auxiliary noise energy, and achieves faster performance recovery following sudden path changes.

Existing early-exit model training paradigms—joint and separate—lack theoretical grounding and systematic empirical evaluation. Method: We formalize training mechanisms into three standardized categories: joint, separate, and hybrid; and propose a staged training strategy: first independently training the backbone, then jointly optimizing the backbone and exit classifiers. We conduct rigorous analysis via information bottleneck theory, loss curvature modeling, numerical rank estimation of activation matrices, and extensive experiments across multiple architectures (ResNet, ViT) and datasets (CIFAR, ImageNet). Contribution/Results: Our study uncovers principled patterns governing how training paradigms interact with model architecture and data characteristics. The hybrid paradigm achieves superior accuracy–latency trade-offs: up to 18% inference speedup on ImageNet with <0.3% top-1 accuracy degradation. This work establishes an interpretable, reproducible foundation for training multi-exit models, bridging theoretical insight with practical performance gains.

Traditional neural network training relies on fixed optimization pipelines, rendering it inflexible in dynamically addressing training instability and anomalies. To address this limitation, we propose the first interactive training framework enabling real-time human–AI collaborative intervention. Our method employs a lightweight control server that integrates expert human directives with AI agent feedback to dynamically adjust hyperparameters, data sampling strategies, and model checkpoints during training. This framework introduces, for the first time, a closed-loop interactive paradigm into neural network training, establishing a scalable human–machine collaboration interface coupled with automated response mechanisms. Experimental results demonstrate significant improvements in training stability, reduced sensitivity to initial hyperparameter configurations, and enhanced real-time responsiveness to user-specified customization requirements. The effectiveness is validated across multiple benchmark tasks.

This work investigates how Transformers dynamically acquire inductive capabilities during in-context learning (ICL), specifically focusing on the role of “inductive heads” in transitioning from local n-gram pattern recognition to modeling long-range dependencies. Method: We combine theoretical approximation analysis, synthetic task training dynamics modeling, attention decomposition, and mixed-objective trajectory tracking across training. Contribution/Results: We formally characterize the generalized inductive head mechanism for the first time, revealing a sharp, non-gradual phase transition—from 4-gram modeling to inductive head emergence—during training. We quantify the layer- and head-specific contributions to long-range dependency capture and demonstrate that inductive heads constitute the core architectural substrate underlying ICL emergence. Our study provides the first full-training-dynamics evidence and an interpretable framework for understanding how large language models dynamically generalize, bridging mechanistic analysis with empirical learning trajectories.

This work addresses the slow convergence and training instability commonly observed in traditional Mixture-of-Experts (MoE) models, which stem from the joint optimization of routing policies and expert weights. To overcome these limitations, the authors propose Grouter, a novel approach that introduces a preset routing mechanism: high-quality routing structures are distilled from a pre-trained MoE model and then fixed, effectively decoupling routing optimization from expert weight updates. Grouter further incorporates expert folding, expert fine-tuning, and structure-prior-guided training strategies to enable efficient adaptation across diverse model configurations and data distributions. Experimental results demonstrate that Grouter improves training data utilization by 4.28× and achieves up to 33.5% higher throughput, significantly enhancing both the efficiency and performance of MoE training.

This work addresses the issue in frozen vision-language models where learnable gating mechanisms are inadvertently disabled by optimizers during training, thereby collapsing auxiliary signal pathways. The authors identify gradient vanishing and negative utility as the underlying causes and propose a fixed-scale injection strategy that eliminates the need for learnable gates. Innovatively integrating entailment cones and angular repulsion on hyperbolic manifolds into LoRA fine-tuning, they regularize the model with a geometric auxiliary loss. During inference, the geometric pathway is retained, preserving relational question-answering accuracy while enhancing attribute-based performance. Notably, on out-of-distribution Visual Spatial Relation (VSR) tasks, the geometric loss stabilizes spatial signals—its removal leads to a 4.6 percentage point drop in performance.

Existing machine unlearning methods often suffer from either over-unlearning, which degrades model utility, or under-unlearning, which leaves residual privacy risks, primarily due to the absence of precise guidance signals. To address this, this work proposes the GSUO framework, which introduces, for the first time, a task-aware, fine-grained, and differentiable guidance mechanism that dynamically adjusts unlearning intensity based on the memorization strength of individual samples. GSUO supports diverse unlearning scenarios, including random subsets and class-level removal. Extensive experiments demonstrate that GSUO significantly outperforms 14 baseline methods in terms of unlearning efficacy, model generalization, and computational efficiency, offering a highly effective, reliable, and versatile solution for machine unlearning.

This work addresses the high inference cost of large audio language models by proposing HeadRouter, a training-free dynamic head-weight routing mechanism that exploits the previously unobserved sparsity and heterogeneity of attention head responses across diverse audio tasks. Recognizing that existing token compression methods overlook the varying importance of different attention heads, HeadRouter introduces a task-adaptive token pruning strategy guided by importance-aware dynamic routing. This approach enables efficient audio sequence compression while preserving or even enhancing model performance. Evaluated on the AudioMarathon and MMAU-Pro benchmarks, HeadRouter achieves 101.8% and 103.0% of the original performance on Qwen2.5-Omni-3B and 7B models, respectively, using only 70% of the original audio tokens.

This study addresses the lack of systematic analysis regarding the upstream sources of steering signals in activation intervention research, which has limited intervention efficacy. By fixing downstream intervention conditions and systematically manipulating source context and activation reading strategies, the work identifies the "execution boundary state" as a critical source of effective steering signals. To enhance signal purity and stability, the authors propose a tail-truncation method that disentangles prompt and continuation semantics. Experiments across three instruction-tuned models and four steering tasks demonstrate that judicious selection of source activations substantially improves intervention performance, with execution boundary states consistently outperforming contexts containing only target behaviors.

This work addresses the challenge that privileged information—available during training but inaccessible at deployment—can mislead models when it is noisy or weakly informative. To mitigate this issue, the authors propose a joint training framework that simultaneously optimizes a teacher model leveraging privileged information and a student model restricted to inputs available at test time. Through an innovative coupling mechanism and an alternating optimization algorithm, the student selectively distills useful knowledge from the teacher while avoiding the propagation of its errors. Theoretical analysis establishes conditions under which this joint training improves accuracy and supports efficient implementation even for high-dimensional, large-scale models. Experiments on both synthetic and real-world datasets demonstrate that the proposed method significantly outperforms conventional two-stage baselines and exhibits robustness to low-quality privileged information.