Cloud Services Software Engineer

NVIDIA•Redmond, WA

2d•$152,000 - $287,500

About The Position

Join NVIDIA's DGX Cloud Team to develop infrastructure that powers AI research. The team optimizes the efficiency and resiliency of machine-learning workloads and builds scalable AI infrastructure tools and services, providing researchers with a stable, scalable environment. As a senior distributed software engineer, you will develop and optimize AI infrastructure services for performance and resilience in DGX Cloud. You will work on backend systems at the intersection of machine learning, distributed systems, and high-performance computing, helping build an AI platform for large-scale training workflows across major cloud service providers and on-premises AI clusters.

Requirements

Bachelor's degree or higher in Computer Science or a related technical field, or equivalent experience.
At least 5 years of hands-on backend development experience, preferably with Python, Go, C/C++, or similar high-performance languages.
A consistent track record of building and scaling large-scale distributed systems.
Experience with cloud computing platforms such as AWS, Azure, and GCP; container technologies such as Docker and Kubernetes; and HPC/AI platforms such as Slurm.

Nice To Haves

Real-world experience with deep-learning frameworks and orchestrators such as PyTorch, TensorFlow, JAX, and Ray.
Experience developing a framework plugin architecture that enables visible integration between a framework and the cluster scheduler.
Strong understanding of NVIDIA GPUs, network technologies, and their failure patterns.
Experience with AI models and AI-based tools.
Code contribution references.

Responsibilities

Develop solutions at the intersection of machine learning, distributed systems, and high-performance computing.
Design, develop, and optimize microservices orchestrated by Kubernetes for large-scale AI training workflows on AI training supercomputers at major cloud service providers.
Co-design and implement APIs that integrate services with NVIDIA's resiliency stack, including telemetry, break/fix automation, checkpoint, and execution systems.
Develop a submission abstraction that enables model engineers and training platforms or frameworks to submit long-running training jobs while concealing infrastructure complexity, automatically restarting jobs after failures, maintaining efficiency, and promptly informing users.
Build modular services that can be coordinated with and deployed on on-premises AI clusters using NVIDIA hardware and cloud services.