model surgery

Fixed-architecture large neural networks suffer from low computational efficiency and poor generalization. Method: This paper proposes an end-to-end differentiable dynamic scalable feedforward network. Its core innovation is the first formulation of network size as a continuous, differentiable variable explicitly embedded into the loss function, enabling joint gradient-based optimization of both architecture and parameters. A single learnable structural control weight governs node addition and removal during training, allowing the network to grow adaptively from a minimal initial structure to a task-optimal size—eliminating the need for post-hoc pruning. The approach integrates scale-aware loss, dynamic computation graph construction, and differentiable structural evolution. Results: Experiments on nonlinear regression and classification tasks demonstrate that the method outperforms static networks of comparable size, automatically converging to smaller, more efficient architectures with superior generalization performance.

Neural networks often suffer significant performance degradation under distributional shifts—such as out-of-distribution generalization, spatiotemporal extrapolation, and cross-domain transfer—due to mismatches between training and deployment data distributions. To address this, we propose a conditional dynamic weight update framework. Its core innovation is a weight anomaly regression mechanism: sensitive weight change patterns induced by distribution shifts are identified via subset retraining; an interpretable regression predictor is then constructed to map input features to weight increments; finally, model parameters are conditionally extrapolated. The method integrates weight difference extraction, regression modeling, and extrapolation techniques, and is empirically validated on multi-source climate observation datasets. Across temporal, spatial, and cross-domain extrapolation tasks, it substantially improves prediction accuracy and robustness on out-of-distribution data, while preserving interpretability and practical applicability.

Deep learning training incurs substantial computational costs, prolonged training time, and high carbon emissions. To address this, this paper proposes a dynamic data reduction method that adaptively prunes input data in real time during image classification training. Unlike static cropping, our approach jointly optimizes data reduction and model accuracy: it evaluates sample importance via gradient sensitivity analysis and integrates online sample selection with adaptive batch-size reduction to dynamically adjust the volume of training data per iteration. Experiments on CIFAR-10, CIFAR-100, and an ImageNet subset demonstrate that the method preserves original model accuracy while reducing both training time and carbon emissions by approximately 50%. This yields significant improvements in training efficiency and environmental sustainability without compromising model performance.

This work addresses catastrophic forgetting in fine-tuning pretrained models, where newly acquired knowledge overwrites previously learned information. To mitigate this issue, the authors propose a function-preserving model expansion approach that mathematically duplicates and scales parameters of selected Transformer submodules during initialization. This technique enables stable training and faithful retention of original model capabilities without altering the initial functionality. By circumventing the traditional trade-off between plasticity and stability, the method achieves performance comparable to full fine-tuning while expanding only a minimal number of layers. Consequently, it fully preserves the model’s original knowledge and substantially reduces computational overhead.

Neural network pruning under data-unavailable scenarios remains challenging, particularly in preserving model robustness without access to original training data. Method: This paper proposes a robustness-preserving pruning framework that operates without any training data. Departing from mainstream fine-tuning–dependent paradigms, it introduces robustness metrics—such as gradient sensitivity and adversarial response—as explicit supervision signals into the data-free pruning pipeline. The method integrates a progressive, conservative pruning strategy with random-optimization–driven channel-level sparsification to jointly optimize both accuracy and open-world robustness. Results: Extensive experiments across multiple CNN architectures demonstrate that our approach achieves an average 12.3% improvement in robust accuracy over state-of-the-art data-free pruning methods, while constraining clean accuracy degradation to within 1.5%. This significantly enhances the practicality and reliability of lightweight models deployed on resource-constrained devices.

This work addresses the challenge that network growth mechanisms in structural plasticity often fail to effectively integrate newly inserted neurons due to weak gradient signals, leading to training instability—particularly pronounced in complex image classification tasks. The study presents the first systematic analysis of the asymmetry between growth and pruning, arguing that growth should be treated as a time-sensitive optimization process and highlighting the critical role of insertion stability for final performance. Building upon a structural plasticity framework with a convolutional backbone, the authors evaluate Grow and Prune strategies on image classification and continual learning benchmarks, introducing targeted interventions concerning optimizer state, unit insertion, and trainability. Experiments demonstrate that while Grow can achieve high accuracy in later stages of structural editing, it only becomes competitive in continual learning when newly added units are granted sufficient time for effective integration.

This work addresses the challenge of efficiently simplifying over-parameterized deep neural networks during training by proposing an online network simplification strategy grounded in training dynamics. The method uniquely integrates the geometric characteristics of neural collapse with layer importance analysis, employing the inverse Fisher criterion as an efficient proxy metric to quantify feature collapse. This enables automatic identification of both the boundary between feature extraction and classification and the optimal timing for simplification. Upon detection, the latter portion of the network is replaced with a lightweight classification head, and training proceeds uninterrupted. Evaluated on image classification tasks across MLP, VGG, and ResNet architectures, the approach substantially reduces model parameters while preserving accuracy comparable to that of the original full-sized models.

This work addresses the diminished understanding of neural network fundamentals caused by the widespread use of high-level deep learning libraries. To bridge this gap, the authors construct a complete neural network framework from scratch, eschewing automatic differentiation and prebuilt modules. The implementation explicitly details forward and backward propagation, incorporates multiple activation functions, L2 regularization, and advanced optimizers such as Adam. Designed to balance pedagogical clarity with engineering scalability, the framework demonstrates numerical stability, correctness, and generalization capability on multiclass classification tasks. It thus provides a reproducible and extensible tool for both research and instruction, fostering deeper insight into the core principles of deep learning.

Neural networks often suffer from neuron deactivation when dynamically expanded, leading to ineffective capacity growth. To address this, we propose the Shared-Weight Expander (SWE) and the Steepest-Gradient Voting Allocator (SVoD). SWE enforces parameter smoothness and inheritance by coupling newly added neurons with existing ones via weight sharing. SVoD dynamically allocates expansion budgets across layers based on gradient magnitude, enabling hierarchical, adaptive expansion in deep networks. Together, SWE and SVoD support end-to-end backpropagation training. Experiments on four benchmark datasets demonstrate that our method significantly mitigates neuron deactivation. Compared to state-of-the-art expansion strategies and baseline models, it consistently improves both accuracy and parameter efficiency. These results validate the architectural scalability and training stability of the proposed approach.

To address the low training efficiency, poor generalization, and strong hyperparameter sensitivity of neural networks across varying scales, this paper proposes a scale-invariant adaptive optimization framework. The method unifies adaptive optimization, second-order information approximation, learning-rate scaling invariance, and gradient compression, thereby decoupling optimization from model size and hardware configuration. Its core innovation lies in a scale-robust update paradigm that ensures stable optimization dynamics under variations in parameter count, batch size, and device count. Extensive experiments across diverse architectures—including MLPs, CNNs, and Transformers—and benchmarks—including CIFAR-10/100, ImageNet, and WikiText—demonstrate that the framework achieves 1.3–2.1× speedup over baseline optimizers, improved convergence stability, significantly reduced hyperparameter sensitivity, and eliminates the need for scale-specific hyperparameter tuning.