Dual-Node NVIDIA DGX Spark over Tailscale: A Remote-Access Testbed for Distributed LLM Training and Cyber-Threat-Intelligence Fine-Tuning

2026-08-07

AI Summary

This study addresses the lack of reproducible practices for distributed large language model (LLM) training and cybersecurity fine-tuning on desktop-scale multi-node hardware. Leveraging two NVIDIA DGX Spark systems interconnected via Tailscale networking and a 200 Gb/s direct fiber link, we present the first end-to-end distributed LLM pretraining and cybersecurity threat intelligence (CTI) fine-tuning on the dual-node Grace Blackwell compact platform. Our system employs PyTorch torchrun with DistributedDataParallel (DDP) and NCCL, integrates a custom security-focused dialogue dataset, and utilizes Ollama-based evaluation models. Within four days, the pipeline processed 653 million tokens (achieving ~1,890 tokens/s throughput), and CTI fine-tuning improved security task scores from 2.06 to 2.29 on a 0–10 scale. The deployment also supports undergraduate AI education and query engine applications, with a fully open-sourced implementation.

Abstract

Compact AI systems make local language-model experimentation increasingly accessible, yet practical evidence for multi-node training on desktop-class accelerators remains limited. This report presents a proof-of-concept deployment of distributed NanoChat pretraining across two NVIDIA DGX Spark systems, each with a GB10 Grace Blackwell system-on-chip and 128 GB of unified memory, administered remotely over a Tailscale mesh VPN and connected for training by a dedicated 200 Gb/s QSFP56 direct fiber link. PyTorch torchrun, DDP, and NCCL were configured with one process per node, a depth-20 NanoChat model, a local batch size of 32 per node, and a 2,048-token context, giving a global batch of 131,072 tokens per step. The run sustained a step time of about 69.4 s (about 1,890 tokens/s), processing about 653 million tokens over four days. We document link configuration, container setup, interface binding, a step-zero evaluation bug that triggered NCCL timeouts, checkpointing, and troubleshooting lessons, as a reproducibility reference for small labs. We also built a cybersecurity fine-tuning dataset from 77 CISA advisories (338 training, 37 validation conversations) and ran a 17-question held-out evaluation comparing a baseline SFT checkpoint against a CTI-augmented checkpoint with an Ollama-hosted LLM judge. CTI-specific categories improved while general-knowledge categories regressed, for a small overall change from 2.06 to 2.29 on a 0-10 scale. The same cluster supports a 400-level AI course (CS 426) and a query engine for CompTIA Security+ POGIL activities in CBS 255, showing modest local infrastructure can serve both research and teaching. The study establishes feasibility rather than a scaling-efficiency claim, since single-node throughput used for comparison was estimated, not measured under matched conditions. Runbook and scripts are available (see Code Availability).

Problem

• distributed LLM training
• multi-node training
• cyber-threat-intelligence fine-tuning
• desktop-class accelerators
• small-scale AI infrastructure

Innovation

• distributed LLM training
• NVIDIA DGX Spark
• Tailscale mesh VPN
• cyber-threat-intelligence fine-tuning
• multi-node desktop AI