training diagnostics

This study addresses the coupled challenges of class imbalance and class overlap in software defect prediction, which jointly impair model training dynamics and performance. The authors propose the first interaction-aware protocol for analyzing training dynamics under these intertwined data quality issues. By training a fixed multilayer perceptron (MLP) under three conditions—imbalance only, overlap only, and their coupling—the protocol systematically records training trajectories. Integrating effect size analysis, sensitivity analysis, and rule-based classification, it constructs a taxonomy of training dynamic patterns. The work uncovers distinctive neural network behaviors specific to the coupled scenario, offering empirical insights and novel diagnostic tools to enhance the understanding, evaluation, and refinement of defect prediction models.

Current AI research often treats models as static artifacts, overlooking the fundamental influence of training dynamics on critical properties such as capability, bias, robustness, and safety. This work proposes shifting the focus toward the training process itself to establish a science of AI centered on training dynamics. By analyzing the interactions among data, objectives, architectures, and optimizers, the paper develops a theoretical framework that is predictive, intervenable, and design-oriented. Integrating approaches from mechanistic interpretability, fairness, memory mechanisms, and simplicity biases, it uncovers causal links between early-training signals and final model behavior. The study systematically outlines key challenges and open problems, offering both theoretical pathways and practical foundations for extending scaling laws beyond performance to encompass multidimensional model attributes.

This study addresses the critical issue of model instability in software engineering optimization, which leads to substantial variability across repeated experiments and undermines both credibility and practical utility. Rather than treating instability as mere random noise, this work conceptualizes it as a quantifiable and manageable property that should be integrated into standard evaluation frameworks. By systematically modulating label usage, model complexity, and partition scoring strategies—combined with multi-objective optimization, causal intervention, data locality analysis, and model calibration—the proposed approach significantly enhances result consistency. Empirical evaluation demonstrates that the optimized configuration reduces the standard deviation of error by 22% on average and outperforms default settings in 119 out of 127 datasets, achieving a 4.8-fold improvement in result consistency.

This study addresses the critical challenge of “silent failures”—erroneous model decisions without system crashes—in production machine learning systems, which undermine conventional monitoring and expose a gap in empirically grounded observability practices. Through seven cross-industry focus group interviews, we applied qualitative thematic coding and scenario mapping to systematically identify the types of observability data practitioners collect and their concrete uses in model validation, anomaly detection, and root-cause diagnosis. Our findings constitute the first empirical characterization of key blind spots in current ML observability tooling: delayed response to feature drift, lack of decision traceability, and difficulty quantifying business impact. Based on these insights, we propose three foundational design principles for next-generation observability tools—explanability-awareness, causal attribution support, and business-impact alignment—and establish an empirically anchored theoretical foundation for future evaluation frameworks and standardization efforts. (149 words)

Traditional neural network training relies on fixed optimization pipelines, rendering it inflexible in dynamically addressing training instability and anomalies. To address this limitation, we propose the first interactive training framework enabling real-time human–AI collaborative intervention. Our method employs a lightweight control server that integrates expert human directives with AI agent feedback to dynamically adjust hyperparameters, data sampling strategies, and model checkpoints during training. This framework introduces, for the first time, a closed-loop interactive paradigm into neural network training, establishing a scalable human–machine collaboration interface coupled with automated response mechanisms. Experimental results demonstrate significant improvements in training stability, reduced sensitivity to initial hyperparameter configurations, and enhanced real-time responsiveness to user-specified customization requirements. The effectiveness is validated across multiple benchmark tasks.

This study addresses the limitations of existing SysML verification approaches, which are often tool-dependent and restricted to performance properties, lacking support for automated validation of behavioral and interface requirements. To overcome these shortcomings, this work proposes a tool-agnostic, automated verification workflow driven by SysML test cases, integrating UML Testing Profile and behavioral diagram constructs to enable unified validation of multidimensional attributes—including behavior, timing, and state responses. The methodology was developed through a mixed-methods research strategy combining literature review and stakeholder interviews, and its efficacy was empirically validated across two independent SysML toolchains. The approach not only transcends the constraints of conventional parametric methods but also enables automatic traceability of verification results back to the original model elements.

This work addresses the lack of a general, auditable dynamic control mechanism in existing training systems, which typically rely on framework-specific code. The authors propose the first cross-framework, open-source control plane that exposes training interfaces through a unified protocol, integrating declarative configuration, request validation, and secure control-point scheduling within the Aim workspace to enable metric monitoring, real-time intervention, and operational traceability. The system supports safe human and automated controller interventions during training while fully logging all operational trajectories. Experiments across five NLP and reinforcement learning tasks demonstrate its effectiveness, and the open-source implementation provides a foundation for reproducible human-in-the-loop training.

This work addresses the challenge that symbolic execution engines involve numerous parameters with complex, interdependent effects, often leading users—due to limited understanding—to rely on suboptimal default configurations, while existing automated tuning approaches lack interpretability. To bridge this gap, the authors propose a human-in-the-loop parameter tuning paradigm and develop Symetra, a visual analytics system that enables dual-perspective overviews of how parameters influence branch coverage. Symetra supports interactive comparison of configuration sets and facilitates pattern recognition. Experimental results demonstrate that expert users leveraging Symetra not only accurately interpret parameter interactions and identify complementary configurations but also achieve significantly higher branch coverage and tuning efficiency compared to fully automated methods, thereby effectively overcoming the interpretability bottleneck in symbolic execution parameter optimization.

This work addresses the challenge of training failures in large language models, which often persist for thousands of optimization steps before manifesting as obvious loss divergence, leading to substantial computational waste. The authors propose a mechanism-aware, proactive monitoring approach that deploys internal detectors at the earliest points where failure signatures become measurable. Specifically, they introduce diagnostic signals grounded in the functional principles of critical modules—such as spectral entropy derived from the bilinear decomposition of QK matrices under low-precision Flash Attention and behavioral metrics of MoE router expert selection. By leveraging these module-specific indicators, the method enables early and accurate identification of diverse failure modes thousands of steps before loss divergence occurs, significantly outperforming conventional detection strategies based solely on loss values or gradient norms, particularly in scenarios involving low-precision attention, excessively high learning rates, or compound faults.

This work addresses the frequent failure of language models in tool calling due to incorrect tool selection, a problem exacerbated by the lack of interpretability and mechanisms for pre-execution intervention. The study reveals, for the first time, that tool selection decisions are concentrated in a specific linear subspace of the model’s output layer, with base models already encoding tool semantics and instruction tuning merely adjusting the output mapping. Through linear probing, activation patching, cosine readout, and mean-difference vector interventions, the authors validate this mechanism across 12 prominent models: those above 4B parameters achieve 93–100% accuracy in precise tool switching, erroneous calls can be anticipated via confidence gaps between tools, and linear probes attain 61–89% top-1 accuracy on 14 aviation-domain tools.