subject-matter expertise

To address the high memory overhead of Mixture-of-Experts (MoE) large language models, which hinders their deployment in resource-constrained settings, this paper proposes the first knowledge distillation framework tailored for MoE teacher models—uniquely exploiting implicit knowledge embedded in non-activated experts. Our method introduces two key components: (1) a Knowledge Augmentation (KA) module that explicitly models latent representations from inactive experts; and (2) a Student-Aware Router (SAR), enabling differentiable joint optimization of the router and student model to facilitate holistic expert knowledge distillation. Extensive experiments across multiple benchmarks demonstrate that the distilled student model achieves substantial parameter reduction while maintaining—and in many cases improving—generation quality, thereby validating the efficacy and superiority of full-expert knowledge distillation in MoE architectures.

Large language model (LLM) agents in multi-agent expert systems suffer from cognitive failures—including bias expression, information drift, and erroneous tool invocation—due to dynamic environmental shifts and misalignment with static evaluation protocols. Method: We propose a dynamic evaluation and context optimization framework featuring a novel ternary diagnostic system that integrates gold-standard annotations, behaviorally mutated silver datasets, and LLM-based agent judges. This system constructs a vectorized improvement suggestion graph for precise cognitive error localization, interpretable root-cause attribution, and cross-instance correction. Contributions/Results: (i) A controlled behavioral-mutation-driven dynamic evaluation protocol; (ii) fine-grained, reproducible performance scoring via LLM agent judges; and (iii) a context repair mechanism grounded in expert behavior transfer. Evaluated on a recruitment assistant system, our approach significantly improves reasoning accuracy and stylistic consistency.

This study investigates the true origins of expert specialization in Mixture-of-Experts (MoE) models, challenging the assumption that routing mechanisms reflect genuine domain-specific expertise. Through theoretical analysis and empirical evaluation, the work demonstrates for the first time that expert usage similarity is entirely determined by the geometric structure of hidden states rather than architecture-induced specialization. It further reveals that load-balancing losses suppress directions corresponding to shared representations, leading to a phenomenon termed “specialization collapse.” Additionally, routing patterns in pretrained MoEs prove largely semantically uninterpretable. Combining linear mapping analysis, geometric modeling of hidden states, cross-model and cross-layer routing comparisons, and theoretical derivation of loss functions, the study validates across five pretrained models a strong alignment between routing behavior and representation-space geometry: expert activations exhibit high overlap even on semantically unrelated inputs, and prompt-level routing fails to predict expert selection during actual inference.

This work addresses routing errors and coverage gaps in complex AI systems caused by overconfident misjudgments of expert models near domain boundaries. It formalizes the human cognitive phenomenon of boundary misjudgment as “Transitive Expert Error” (TEE) and maps it onto AI architectures. By introducing multi-expert activation with disagreement detection at the routing layer, boundary-aware calibration at the expert layer, and coverage gap detection during training, the approach transforms an otherwise unobservable cognitive black box into a monitorable and intervenable system design problem. The method effectively identifies TEE signals—such as anomalous routing decisions and decoupling between confidence and accuracy—across diverse architectures including Mixture-of-Experts (MoE), multi-model orchestration, tool-augmented agents, and retrieval-augmented generation (RAG), significantly reducing hallucinatory outputs in cross-domain tasks.

This work investigates the opaque expert specialization mechanism in Mixture-of-Experts (MoE) models, which limits inference and memory efficiency. By analyzing domain-specific routing patterns and employing an early-decoding framework, the study systematically examines how individual experts contribute to model outputs. Through comprehensive analyses—including routing distribution statistics, cosine similarity of hidden states, comparisons between single-expert and ensemble outputs, and perplexity evaluation—the authors find that a small subset of experts handles over 50% of all requests. Remarkably, outputs from a single dominant expert exhibit high consistency with the full model (cosine similarity up to 0.95), with only a 5% increase in perplexity. These findings suggest that precise expert pruning can substantially enhance inference efficiency without compromising performance, offering a promising avenue for efficient MoE deployment and knowledge localization.

This study investigates how teaching interactions between experts and novices enhance learning agent performance compared to pure expert demonstrations. By constructing a synthetic expert-novice interaction dataset in a spatial navigation task and employing a Transformer-based architecture with controlled experimental designs, the work systematically evaluates learning efficacy and generalization under varying data conditions. For the first time within a controlled paradigm, it demonstrates that cognitive discrepancies encoded through multi-agent identities significantly improve model robustness, enabling emergent expert-level performance even when expert behaviors are sparse. The results consistently show that models trained on teaching interaction data outperform baseline approaches relying solely on expert demonstrations across diverse scenarios.

This work addresses the lack of interpretability and controllability in expert specialization mechanisms within sparse Mixture-of-Experts (MoE) models. The authors propose a low-dimensional geometric routing method based on cosine similarity, which endows each rank-1 expert with inherent mono-semanticity. For the first time, expert-level specialization is established as a fundamental unit of interpretability, exhibiting structural mono-semanticity, causal verifiability, and zero-overhead inference controllability. Experiments demonstrate that 15% of experts can be explicitly mapped to 10 distinct semantic categories. Furthermore, causal interventions enable significant modulation of target class output probabilities—e.g., increasing temporal category likelihood by 321% or decreasing geographic category likelihood by 23%—with effects that accumulate across multiple layers.

This work addresses a critical limitation in current personalization approaches for large language models (LLMs), which often overlook the heterogeneity of users’ domain-specific expertise, thereby risking overreliance on AI in areas where users are less competent and potentially inducing professional drift. To mitigate this, the authors propose a scaffolding intervention framework grounded in user competency profiling, dynamically modulating LLM outputs by categorizing domains into strong, mixed, and weak proficiency zones. The framework introduces, for the first time, a structured competency-profile-driven intervention mechanism that integrates typological competency partitioning, competency-conditioned response strategies, and multi-LLM ensembling. Evaluated on MMLU subsets, the approach demonstrates that swapping user profiles can invert performance categories, while selective activation in mixed zones significantly enhances human-AI collaboration reliability.

This work systematically investigates the interplay among key design dimensions in Mixture-of-Experts (MoE) architectures—such as the number of experts, expert granularity, heterogeneity, shared experts, and load balancing—through over 2,000 large-scale pretraining experiments. The study reveals that the number of experts and their granularity are the dominant factors governing model performance, while other design choices exert comparatively limited influence. Notably, increasing the total MoE parameters consistently enhances performance across all active parameter budgets, and the optimal expert size is determined solely by the number of active parameters. Furthermore, the effectiveness of dropless routing is empirically validated, demonstrating consistent performance gains.

This work addresses the limitation of existing AI explanation methods, which typically rely on static user models and fail to accommodate the diverse cognitive strategies and domain knowledge of scientific experts. To overcome this, we propose a novel framework that integrates knowledge graph path reasoning with reinforcement learning–driven agent-based persona modeling to dynamically generate adaptive scientific explanations tailored to individual expert preferences. Our approach enables scalable personalized explainability without requiring extensive human feedback, substantially reducing reliance on expert-annotated data. Evaluated on a drug discovery task, the method achieves state-of-the-art predictive performance while producing persona-driven explanations that domain experts significantly prefer over non-adaptive baselines, with human feedback requirements reduced by two orders of magnitude.