Senior Software Engineer, Resilience Engineering - DGX Cloud

NVIDIA•Santa Clara, CA

3d•$184,000 - $356,500

About The Position

NVIDIA is seeking a Senior Software Engineer for Resilience Engineering within DGX Cloud. The role will help shape how DGX Cloud approaches reliability and operational excellence, drawing on experience running large-scale systems and applying rigorous operational practices. The engineer will build reliability strategies and tools, improve production systems, and help teams strengthen resilience in a 24/7 environment.

Requirements

Deep hands-on experience running large-scale production systems, with a proven track record.
Detailed understanding of failure modes in large systems, including cascading dependencies and retry storms.
Strong software engineering skills and current hands-on experience with Go, Python, or similar languages.
Proven experience establishing and maintaining an SLO program with operational rigor.
Practical experience in reliability disciplines such as chaos engineering and failure injection.
Ability to influence across team boundaries through credibility and expertise.
10 or more years of industry experience with a bachelor's or master's degree, or equivalent experience operating systems at scale.

Nice To Haves

Experience in a world-class reliability function such as Google SRE or Meta production engineering.
Expertise operating GPU, HPC, or AI training infrastructure and understanding its failure modes.
A track record of measurable reliability improvements within an organization.
Proficiency with modern observability and operational tools such as Prometheus, OpenTelemetry, Grafana, PagerDuty, and Rootly.

Responsibilities

Build an organization-wide reliability strategy and guide the maturation of operational practices in a 24/7 environment.
Establish and maintain a rigorous SLO program and high standards across teams.
Lead response to high-severity incidents, ensuring low-drama, high-signal resolution.
Build and improve production code daily, enhancing the data platform and related tooling.
Implement chaos engineering, failure injection, and resilience testing.
Improve standards through hands-on example and leadership.