Cloud Services Software Engineer

NVIDIA•Redmond, CA

About The Position

NVIDIA’s DGX Cloud Team develops infrastructure that powers AI research, with a focus on improving the efficiency and resiliency of machine-learning workloads and building scalable AI infrastructure tools and services. The team aims to provide AI researchers with a stable, scalable environment and the resources needed to advance innovation. The Senior Software Engineer will develop and optimize AI infrastructure services for DGX Cloud, contributing to cloud-services architecture and the resilience stack. Working in a dedicated backend-development team, the engineer will build infrastructure and products that support NVIDIA’s AI platforms and enable AI research. The role focuses on solutions spanning machine learning, distributed systems, and high-performance computing. It includes developing Kubernetes-orchestrated services for large-scale AI training workflows on training supercomputers at major cloud service providers, designing APIs that integrate with NVIDIA resilience services, and creating abstractions for submitting long-running training jobs. These systems should manage job lifecycles, automatically restart jobs after failures, maintain efficiency, and provide timely information to users. Services should be modular and suitable for coordination with and deployment on on-premises AI clusters using NVIDIA hardware and cloud services.

Requirements

Bachelor’s degree or higher in Computer Science or a related technical field, or equivalent experience.
At least 5 years of hands-on backend development experience, preferably with Python, Go, C/C++, or similar high-performance languages.
A consistent track record of building and scaling large-scale distributed systems.
Experience with cloud platforms such as AWS, Azure, and GCP; container technologies such as Docker and Kubernetes; and HPC/AI platforms such as Slurm.

Nice To Haves

Real-world experience with deep-learning frameworks and orchestrators such as PyTorch, TensorFlow, JAX, and Ray.
Experience developing a framework plugin architecture that integrates a framework with the cluster scheduler in a way visible to users.
Strong understanding of NVIDIA GPUs, network technologies, and their failure patterns.
Experience with AI models and AI-based tools.
References to code contributions.

Responsibilities

Develop solutions at the intersection of machine learning, distributed systems, and high-performance computing.
Design, develop, and optimize Kubernetes-orchestrated microservices for large-scale AI training workflows on AI training supercomputers at major cloud service providers.
Co-design and implement APIs that integrate services with NVIDIA resilience stacks, including telemetry, break/fix automation, checkpoint, and execution systems.
Create a submission abstraction that enables model engineers and training platforms or frameworks to submit long-running training jobs while masking infrastructure complexity.
Support training-job lifecycles, including automatic restarts after failures, efficiency, and timely user notifications.
Build modular services that can be coordinated with and deployed on on-premises AI clusters using NVIDIA hardware and cloud services.