deep learning implementation

Poor reproducibility of deep learning results severely undermines their reliability and verifiability, primarily due to environmental heterogeneity, dependency incompatibilities, closed data/code, opaque workflows, and uncontrolled stochasticity. To address this, we propose the first systematic framework explicitly designed for reproducibility in deep learning. Our approach is grounded in empirical case studies to identify recurrent reproducibility patterns and anti-patterns; incorporates sensitivity analysis to quantify performance variations induced by critical factors—including library versions, hardware platforms, and random seeds; and integrates Docker-based containerization, deterministic dependency pinning, end-to-end provenance tracking, and comprehensive pipeline documentation. Experimental evaluation demonstrates a substantial improvement in reproduction success rates across diverse models and tasks. Furthermore, we distill a generalizable best-practice checklist for deep learning reproducibility—bridging the gap between academic research and industrial practice.

This study investigates the marginal benefits of a three-stage strategy—self-supervised pretraining, intermediate fine-tuning, and downstream task adaptation—for small-scale Vision Transformers (~5M parameters). Motivated by the observation that intermediate fine-tuning may degrade downstream performance due to task misalignment, we propose a systematic ablation framework to assess the impact of varying dataset and objective combinations across stages. Experiments reveal that targeted pretraining substantially improves small-model performance, whereas introducing semantically distant intermediate tasks yields no gain—and often harms performance while wasting compute. The core contribution is the empirical demonstration that, for small ViTs, **the quality of data selection is far more critical than the number of stacked tasks**, challenging conventional assumptions about multi-stage transfer. This finding provides key empirical evidence and methodological guidance for designing efficient, lightweight self-supervised learning paradigms.

This work addresses the challenge of spatial controllability in image generation models, systematically investigating the unified modeling capability of Transformer-based diffusion, flow, and autoregressive architectures under fine-grained spatial conditions—such as edge maps and pose keypoints. We propose control-token pre-filling as an efficient, general-purpose baseline; identify classifier-free guidance scaling and softmax truncation as critical for improving control consistency; and re-validate adapter-based fine-tuning for mitigating task forgetting. Experiments on ImageNet demonstrate that our approach significantly enhances controllability consistency while preserving high-fidelity generation under data-limited regimes. By decoupling the effects of architecture, training methodology, and guidance strategies, this study establishes a reproducible benchmark framework and provides practical design principles for controllable image synthesis.

To address excessive computational overhead when deploying vision models on resource-constrained devices, this paper proposes an efficient Vision Transformer (ViT) architecture design framework. Methodologically: (1) it optimizes the input-output data pathway to enhance representational capacity of lightweight models; (2) it restructures the context window of computationally constrained attention mechanisms to improve local-global modeling efficiency; and (3) it leverages the invertibility and explicit probabilistic modeling properties of normalizing flows to enable high-fidelity, low-overhead knowledge distillation. Experiments demonstrate that the proposed approach achieves comparable or superior accuracy on benchmarks such as ImageNet, while requiring significantly fewer parameters and FLOPs. It also substantially reduces inference latency and memory footprint. The framework establishes a scalable new paradigm for efficient visual understanding at the edge.

High barriers to adopting pre-trained models and a lack of empirical guidance for strategy selection hinder practical deployment in few-shot image classification and object detection. Method: We systematically compare linear probing versus fine-tuning across ResNet, MobileNet, and EfficientNet, and propose an end-to-end TensorFlow framework integrating multi-scale feature-space visualization (PCA, t-SNE, UMAP) to unify analysis of representation evolution. Contribution/Results: Linear probing significantly outperforms fine-tuning under extreme data scarcity (≤100 samples per class) while accelerating training by 3–5×. The framework enables high-accuracy, rapid deployment (<1 hour for fine-tuning) on standard benchmarks (ImageNet-1K, CIFAR-100), balancing beginner-friendly usability with expert-level extensibility. It bridges the gap between theoretical representation analysis and real-world engineering practice.

This work investigates whether the standard Transformer architecture is universally optimal across all tasks and proposes an architectural refinement that introduces task-specific inductive biases through learnable nonlinear components, such as GeLU or softmax. The approach preserves the original Transformer structure while replacing key activation functions with task-optimized counterparts learned during training. Experimental results demonstrate that this modification substantially improves learning speed, in- and out-of-distribution generalization, and training stability on algorithmic reasoning tasks. Consistent, albeit more modest, performance gains are also observed in language and code modeling, accompanied by enhanced cross-domain transfer capabilities. These findings indicate that the standard Transformer is not locally optimal for specific tasks and that incorporating task-tailored design elements—despite a trade-off in generality—can effectively boost performance.

Training high-capacity vision models is computationally expensive, and existing model-growing approaches rely on pre-trained narrow models, failing to accurately reflect the total training cost. This work proposes a Recursive Block-Diagonal Coupling (RBDC) training protocol that recursively integrates independently trained narrow models into a high-performance wide model via a parameter-free block-diagonal structure. The method incurs no additional parameters and substantially reduces training FLOPs—achieving a 30% reduction in computational cost on ImageNet compared to standard from-scratch training while maintaining comparable accuracy. Under identical computational budgets, RBDC outperforms current model-growing strategies and serves as a superior backbone, enhancing performance in downstream tasks such as object detection and instance segmentation.

This work addresses the high computational cost of conventional fine-tuning for instance segmentation, which typically requires updating a large fraction (40–55%) of parameters in large pre-trained models. To improve parameter efficiency, the study explores parameter-efficient fine-tuning (PEFT) methods, introducing LoRA into deformable attention mechanisms for the first time and systematically evaluating the trade-offs between performance and efficiency based on the number and placement of adapters within the Transformer architecture. Experimental results demonstrate that by fine-tuning only 1–6% of the model parameters, the proposed approach matches or even surpasses the performance of full fine-tuning across four benchmark datasets. These findings validate the efficacy and feasibility of PEFT for instance segmentation and further reveal that its effectiveness is influenced by dataset complexity and model architecture.

This study investigates how linear recurrent Transformers equipped with layer normalization implicitly learn the power method through gradient descent when trained on principal component prediction tasks. The work reveals an “algorithmic implicit bias”: in the absence of explicit supervision, the self-attention layers automatically converge to solutions that implement power iterations, with each layer corresponding to one update step of the power method. Theoretical analysis demonstrates that layer normalization is essential for realizing the exact power method—models without it fail to replicate the algorithm, resulting in significantly degraded performance. This paper is the first to establish the pivotal role of layer normalization in inducing algorithmic inductive biases and provides provable guarantees for the resulting performance gap.

Existing training monitoring approaches struggle to diagnose under-optimization in individual Transformer layers under low-bit (including binary) quantization, as global loss metrics fail to capture layer-wise learning dynamics. This work proposes a layer-wise peeling framework that constructs lightweight reference solutions for each layer, integrating intermediate representation reconstruction, local optimization, and multi-permutation projection to establish the first practical layer-level performance benchmark. The method enables early identification of “pseudo-converged” layers during training, and the derived reference bounds match or even surpass the original model’s performance across diverse quantization settings, substantially enhancing fine-grained diagnostic capability throughout the training process.