Senior System Software Engineer, DGX Cloud AI Infrastructure

NVIDIA•Santa Clara, CA

About The Position

NVIDIA’s DGX Cloud team builds infrastructure for large-scale AI training, supporting NVIDIA’s own workloads and those of its customers. The Senior System Software Engineer will join the Recovery and Resilience team and own AI infrastructure services, tools, and open-source projects that make training more resilient and efficient, from design through production. The role also includes mentoring engineers and shaping how the team operates and delivers its work. The engineer will build resilient AI infrastructure on Kubernetes and Slurm for supercomputers at major cloud providers and on-premises clusters. This includes APIs connecting to NVIDIA resiliency systems, a job-submission layer that automatically restarts long-running training jobs after infrastructure failures, and agentic tools that help customers find bottlenecks, diagnose failures, and recommend fixes. The role also drives open-source releases and collaboration with customers and the community, and establishes service support, monitoring, alerts, CI/CD, integration-test gates, and on-call plans.

Requirements

BS, MS, or PhD in Computer Science or a related field, or equivalent experience.
12+ years of experience.
Proven experience taking large systems from design through production.
Strong experience with Kubernetes and/or Slurm at scale.
Hands-on experience with PyTorch, TensorFlow, JAX, or Ray.
Strong understanding of NVIDIA GPUs and networking, including their failure modes.
Experience running production services, including CI/CD, monitoring, alerting, and on-call.

Nice To Haves

Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers.
Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting.
Background building external services or maintaining open-source projects in ML infrastructure.

Responsibilities

Drive the design, build, launch, and operation of services, tools, and open-source projects.
Plan and assign team work, mentor junior engineers, and review designs.
Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA and customer workloads, including cloud-provider supercomputers and on-premises clusters.
Build APIs that connect to resiliency systems for telemetry, break/fix automation, checkpointing, and execution.
Build a job-submission layer that shields users from infrastructure failures and automatically restarts long-running training jobs.
Build agentic tools to help customers improve training efficiency by identifying bottlenecks, diagnosing failures, and recommending fixes.
Drive open-source work, release components for customer and community use, and collaborate with users to improve them.
Establish service support, monitoring, alerts, CI/CD, integration-test gates, and on-call plans.