pytorch implementation

This work addresses the lack of transparent, scalable, and deeply PyTorch-integrated open-source tools for post-training large language models, which hinders research iteration and deployment efficiency. We propose a native PyTorch-based, modular post-training framework centered on the principle of “hackability,” offering composable model builders, training recipes, and a distributed training stack that support diverse fine-tuning strategies and hardware configurations. While maintaining high performance and memory efficiency, the framework significantly enhances code transparency and research flexibility. Empirical evaluations demonstrate that it matches or even surpasses mainstream tools such as Axolotl and Unsloth across multiple post-training scenarios, thereby facilitating efficient and reproducible scientific exploration.

To address inefficient backpropagation and excessive memory overhead in small-scale computational graph training on single-CPU nodes, this paper proposes BurTorch, a lightweight training framework. Methodologically, BurTorch abandons the abstraction-layer bloat of general-purpose deep learning frameworks and instead embraces the fundamentals of compiled-language programming and numerical optimization: it employs hand-optimized C++, static computational graphs, explicit memory management, and a native backpropagation engine grounded in the Linnainmaa/Rumelhart chain rule. This system-level design directly targets performance bottlenecks inherent to small-graph training. Experiments demonstrate that, on representative small-graph tasks, BurTorch achieves up to 2000× speedup and 3500× memory reduction over PyTorch; even on a micro-scale GPT-3 model, it delivers 20× acceleration and 80× memory savings. These results significantly advance efficient, resource-constrained deep learning training.

This work addresses the challenge of deploying mainstream deep learning frameworks in resource-constrained environments, where their large size and lack of lightweight yet fully featured alternatives pose significant limitations. To this end, we propose and implement a lightweight tensor computation library built in Rust, leveraging its performance and memory safety guarantees to construct an efficient computational engine. The system exposes a PyTorch-like Python interface via PyO3, supporting essential features including n-dimensional tensors, dynamic computation graphs, reverse-mode automatic differentiation, neural network layers, and optimizers. The resulting installable package occupies only a few megabytes—orders of magnitude smaller than PyTorch or TensorFlow—while retaining the core capabilities necessary for research and development on CPU-based systems.

Industrial-scale recommendation and ranking models feature highly complex and continuously evolving architectures, rendering traditional optimization approaches—based on manual intervention or module-level rules—difficult to scale. This work proposes the first extensible and customizable operator-level automatic transformation framework integrated into PyTorch 2.x. By leveraging FX intermediate representation, the PT2 compiler, predefined pattern matching, and a greedy search algorithm, the framework achieves general-purpose model optimizations while strictly preserving computational semantics. Evaluated on real-world industrial recommendation models, the approach delivers up to 63% inference speedup, a 6% reduction in peak memory usage, and over 400 seconds of compilation time savings. The implementation has been open-sourced as part of PyTorch 2.x.

To address the lack of automated, systematic tools for hyperparameter optimization in neural operators, this paper proposes the first fully automated, parallelized hyperparameter search framework specifically designed for neural operators. Built upon PyTorch and integrated with Ray-Tune for distributed optimization, the framework supports joint hyperparameter tuning across diverse operator architectures (e.g., Fourier-based and CNN-based), datasets, and spatial/temporal resolutions, while enforcing a fixed parameter-count constraint during training. It provides extensible, modular interfaces that balance usability and customization, enabling user-defined models and datasets. Evaluated on multiple PDE benchmark tasks—including Burgers’, Darcy, and Navier–Stokes equations—the framework achieves state-of-the-art performance, demonstrating its efficacy in modeling real-world physical data and modern neural operator architectures. By significantly reducing implementation complexity and experimental overhead, it lowers the barrier to entry for neural operator research and deployment.

This work addresses the diminished understanding of neural network fundamentals caused by the widespread use of high-level deep learning libraries. To bridge this gap, the authors construct a complete neural network framework from scratch, eschewing automatic differentiation and prebuilt modules. The implementation explicitly details forward and backward propagation, incorporates multiple activation functions, L2 regularization, and advanced optimizers such as Adam. Designed to balance pedagogical clarity with engineering scalability, the framework demonstrates numerical stability, correctness, and generalization capability on multiclass classification tasks. It thus provides a reproducible and extensible tool for both research and instruction, fostering deeper insight into the core principles of deep learning.

This work proposes a training-free method for constructing neural networks that precisely simulate arbitrary Turing machines. Given a formal description of a Turing machine, the system automatically compiles it into a corresponding neural network whose forward pass exactly mirrors a single step of the machine’s execution. Grounded in first principles, the approach employs ReLU networks to implement Boolean logic and adders, leverages Cantor set encoding together with hard attention mechanisms to manage tape read–write operations, and demonstrates Turing completeness within both Transformer architectures—incorporating self-attention and cross-attention—and recurrent neural network frameworks. This study establishes a reproducible and verifiable theoretical and practical foundation for integrating neural and symbolic computation.

This work addresses the limitations of conventional auto-schedulers, which treat candidate schedules as static code snapshots and ignore dependencies among scheduling actions, leading to sensitivity to superficial changes and inefficient search. To overcome this, the paper introduces— for the first time—a world model into compiler optimization, proposing an action-conditional latent dynamics model that represents the scheduling process as an evolution of program states in a continuous latent space, thereby avoiding repeated AST modifications and re-encodings. Integrated into TVM AutoScheduler, the approach combines a lightweight transition network with a hardware-aware ranking mechanism. Experiments show that, under the same 64 measurements, it outperforms Ansor by 1.37× on GPU and 1.54× on CPU; with only one-tenth the measurement budget, it achieves 97.8% of Ansor-10K’s performance; and full-model inference runs 4.61× and 3.67× faster than PyTorch and PyTorch-opt, respectively.

This work addresses the challenge of defect detection in deep learning libraries such as TensorFlow and PyTorch, where complex APIs often lead to subtle bugs and existing testing approaches suffer from high false-positive rates due to imprecise specifications. To overcome this limitation, the authors propose a machine learning classifier that leverages tensor shape abstraction as a precise input representation for API validity constraints. By integrating runtime feedback to automatically generate labeled training data, the method learns accurate usage patterns without relying on manual annotations. Implemented within the ACETest framework, the approach achieves over 91% classification accuracy across 183 APIs and significantly improves test pass rates—from 29% to 61%—demonstrating enhanced precision and scalability in testing deep learning libraries.

Existing benchmarks for code generation struggle to balance task difficulty with the scalability of automated evaluation. This work proposes TensorBench, a benchmark comprising 199 challenging tasks built upon an extended PyTorch compiler framework that supports both dense and sparse tensors. It enables fully automated verification of patch correctness through comprehensive test suites and introduces, for the first time, an agent-driven self-augmenting testing mechanism. Integrating compiler techniques, tensor optimizations, IR transformations, and randomized regression testing, TensorBench effectively evaluates model performance on realistic, system-level programming tasks. Evaluations of seven prominent code-generation agents reveal pass rates ranging from 22.1% to 64.8%, with low inter-task result overlap (Cohen’s κ as low as −0.07), demonstrating the benchmark’s high discriminative power and task diversity.