About the job
NVIDIA is hiring an NCX Senior Engineer who is passionate about NVIDIA Cloud Partner (NCP) infrastructure operations to join our DSX team. This role involves working closely with strategic NVIDIA Cloud Partners to build and improve the operational capabilities essential for running large-scale NVIDIA accelerated infrastructure reliably in production.
Responsibilities
Lead NCP Day 2 operational readiness efforts and collaborate with partners to set up systems, procedures, automation, and operational methods for managing NVIDIA accelerated infrastructure.
Build continuous infrastructure validation methods to validate GPU, CPU, storage, and network health across large-scale AI clusters.
Establish observability and operational telemetry including monitoring, alerting, dashboards, and operational signals across compute, GPU, networking, storage, Kubernetes, and AI workloads.
Develop automated detection and remediation workflows to detect, isolate, drain, repair, validate, and return unhealthy infrastructure to service.
Refine fleet lifecycle administration including NVIDIA driver and firmware lifecycle administration, Kubernetes node maintenance, OS patching, configuration management, upgrades, and configuration drift identification.
Operationalize NVIDIA reference architectures by translating requirements into production operating practices, validation criteria, runbooks, automation, and measurable operational standards.
Qualifications
Minimum
BS, MS, or Ph.D. in Computer Science, Computer/Electrical Engineering, or a related technical field, or equivalent experience.
8+ years of experience in infrastructure engineering, Site Reliability Engineering, DevOps, cloud platform engineering, systems engineering, or similar roles supporting large-scale production environments.
Strong experience operating Linux-based distributed systems and cloud infrastructure in production.
Deep understanding of Kubernetes, containers, cluster scheduling, and the operational lifecycle of large multi-node environments.
Strong understanding of production observability, including metrics, logging, alerting, dashboards, health checks, and operations guided by service level agreements.
Experience crafting automation for infrastructure lifecycle management, failure detection, remediation, upgrades, and configuration management.
Strong networking fundamentals and experience troubleshooting complex distributed systems across compute, network, and storage layers.
Programming and automation experience using Python, Go, shell scripting, or similar languages.
Preferred
Experience managing extensive GPU or accelerated computing infrastructure that supports AI training and inference workloads.
Experience with NVIDIA technologies including DGX/HGX systems, CUDA, NVLink/NVSwitch, NVIDIA networking, InfiniBand, RoCE, GPU Operator, Network Operator, or related NVIDIA infrastructure software.
Proven experience collaborating with NVIDIA Cloud Partners, hyperscale cloud providers, managed AI clouds, or extensive service-provider infrastructure and operating SLOs for large-scale compute infrastructure.
Extensive knowledge of infrastructure observability tools including Prometheus, Grafana, OpenTelemetry, Alertmanager, and scalable telemetry pipelines.
Knowledge of failure modes related to large distributed AI workloads and the infrastructure features necessary to consistently support extended training and production inference.