Senior System Software Engineer, DGX Cloud AI Infrastructure
About The Position
NVIDIA’s DGX Cloud team builds infrastructure for large-scale AI training workloads, including NVIDIA’s own workloads and those of its customers. The Senior Software Engineer will join the Recovery and Resilience team and own AI infrastructure services, tools, and open-source projects that support resilient and efficient training from design through production. The role also involves mentoring engineers and shaping how the team operates and delivers its work. The position focuses on building resilient AI infrastructure on Kubernetes and Slurm for supercomputers at major cloud providers and on-premises clusters. It includes integrating with NVIDIA resiliency systems, developing a job submission layer that automatically recovers long-running training jobs from infrastructure failures, and building agentic tools to identify training bottlenecks, diagnose failures, and recommend fixes. The engineer will also lead open-source contributions and establish operational practices for services, including support, monitoring, alerts, CI/CD, integration test gates, and on-call plans.
Requirements
- BS, MS, or PhD in Computer Science or a related field, or equivalent experience.
- 12 or more years of experience.
- Proven experience taking large systems from design through production.
- Strong experience with Kubernetes and/or Slurm at scale.
- Hands-on experience with PyTorch, TensorFlow, JAX, or Ray.
- Strong understanding of NVIDIA GPUs and networking, including failure modes.
- Experience running production services, including CI/CD, monitoring, alerting, and on-call.
Nice To Haves
- Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers.
- Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting.
- Background building external services or maintaining open-source projects in ML infrastructure.
Responsibilities
- Drive the design, development, launch, and operation of services, tools, and open-source projects.
- Plan and assign team work, mentor junior engineers, and review designs.
- Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA and customer training workloads across cloud-provider supercomputers and on-premises clusters.
- Build APIs that connect to NVIDIA resiliency systems, including telemetry, break/fix automation, checkpointing, and execution systems.
- Build a job submission layer that hides infrastructure failures from users and automatically restarts long-running training jobs.
- Build agentic tools that help customers improve training efficiency by finding bottlenecks, diagnosing failures, and recommending fixes.
- Drive open-source work by releasing components for customers and the community and collaborating with them to improve those components.
- Establish service operating practices, including support, monitoring, alerts, CI/CD, integration test gates, and on-call plans.
Benefits
- Equity
- Benefits
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
