Senior Software Engineer, Distributed Systems Engineer - DGX Cloud

NVIDIA•Us, NC

About The Position

NVIDIA is hiring experienced software engineers to help scale up its AI Infrastructure. We expect you to have significant software engineering experience with cluster operations, operator development, node health monitoring and working with GPU resource scheduling. We welcome out-of-the-box thinkers who can provide new ideas with strong execution bias. Expect to be constantly challenged, improving, and evolving for the better. You will help advance NVIDIA's capacity to build and deploy leading infrastructure solutions for a broad range of AI-based applications. If you're creative, passionate about GPUs, and love having fun, please apply today! For two decades, we have pioneered visual computing, the art and science of computer graphics. With the invention of the GPU - the engine of modern visual computing - the field has expanded to encompass video games, movie production, product design, medical diagnosis and scientific research. Today, we stand at the beginning of the next era, the AI computing era, ignited by a new computing model, GPU deep learning. What you will be doing: You will be part of an DGX Cloud team responsible for production systems that enable large scalable GPU clusters to be used for a variety of AI workloads. Designing and developing a massively distributed scalable platform which would be used to identify, diagnose and remediate non-performant GPU assets. Working with teams across NVIDIA to ensure production AI clusters run reliability and consistently with maximum performance. Evaluating system failures and improving services based on a well-defined incident management process.

Requirements

Direct experience in a software engineering role within a highly technical organization with demonstrable impact from your work.
Highly motivated with strong communication skills, you can work successfully with multi-functional teams, principles, and architects and coordinate effectively across organizational boundaries and geographies.
5+ years in similar role and experience on large-scale production systems.
Experience with common software engineering principles, tools and techniques.
You possess a BS in Computer Science, Engineering, Physics, Mathematics or a comparable Degree or equivalent experience.
Technical knowledge, including a systems programming language (Go, Python) and a solid understanding of data structures and algorithms.

Nice To Haves

Technical competency in managing and automating large-scale distributed systems independent of cloud providers.
Advanced hands-on experience and deep understanding of cluster management systems (Kubernetes, Slurm, Base Command Manager).
Prior experience in asynchronous workflows and/or event driven architecture.
Proven operational excellence in maintaining reliable and performant infrastructure.

Responsibilities

Designing and developing a massively distributed scalable platform which would be used to identify, diagnose and remediate non-performant GPU assets.
Working with teams across NVIDIA to ensure production AI clusters run reliability and consistently with maximum performance.
Evaluating system failures and improving services based on a well-defined incident management process.