About the job
At Netflix, our mission is to entertain the world. We are looking for a senior engineer with deep expertise in distributed model training and the systems required to operate it at scale. You will help shape the architecture of our training platform, improve performance and reliability for large training jobs, and build intuitive platform experiences for ML engineers. Our infrastructure is built on Kubernetes, Ray clusters, and PyTorch distributed training primitives.
Responsibilities
Design and build the platform that powers large-scale machine learning model training, fine-tuning, model transformation and evaluations workflows and use cases from the entire company
Co-design and optimize the systems and models to scale up and increase the cost-effectiveness of machine learning model training
Design easy-to-use APIs and interfaces for experienced ML practitioners, as well as non-experts to easy access the training platform
Qualifications
Minimum
Design, build, and operate platform infrastructure, libraries, and SDKs for large-scale model training. Enable reliable and efficient training workflows for foundation models and generative AI models of all sizes.
Diagnose and optimize the performance of large distributed training jobs, including GPU utilization, memory efficiency, communication overhead, data loading, checkpointing, fault tolerance, and cluster utilization.
Experience with cloud computing providers, preferably AWS
Comfortable with ambiguity and working across multiple layers of the tech stack to execute on both 0-to-1 and 1-to-100 projects
Adopt and promote best practices in operations, including observability, logging, reporting, and on-call processes to ensure engineering excellence.
Excellent written and verbal communication skills
Comfortable working in a team with peers and partners distributed across (US) geographies & time zones.
Preferred
Understand modern and real-world Machine Learning model development workflows and experience partnering closely with ML modeling engineers. Lead technical design reviews, facilitate cross-functional discussions, communicate tradeoffs clearly, and align stakeholders on platform direction and execution priorities.
Familiarity with cloud-based AI/ML services (e.g., SageMaker, Bedrock, Databricks, OpenAI, etc.)
Familiarity with distributed training performance analysis tools and techniques, such as PyTorch Profiler, NVIDIA Nsight Systems, GPU telemetry, communication profiling, or cluster-level utilization analysis.
Experience with large-scale distributed training and different parallelism techniques for scaling up training, such as FSDP and tensor/pipeline parallelism
Expertise in the area of Generative AI, specifically when it comes to training foundation models, fine-tuning them, and distilling them to smaller models