Senior Software Engineer, Resilience Engineering - DGX Cloud
About The Position
NVIDIA is seeking a Senior Software Engineer for Resilience Engineering on its DGX Cloud team. The role focuses on redefining reliability and operational excellence for DGX Cloud, building organization-wide reliability strategy, and maturing operational practices in a 24/7 environment. The engineer will establish rigorous service-level objective (SLO) practices, lead high-severity incident response, improve production code and data-platform tooling, and implement chaos engineering, failure injection, and resilience testing. The role requires hands-on experience operating large-scale production systems, strong software engineering skills, and the ability to influence teams through technical credibility and leadership.
Requirements
- Deep, hands-on experience running large-scale production systems, with a proven track record.
- Detailed understanding of failure modes in large systems, including cascading dependencies and retry storms.
- Strong software engineering skills and current, hands-on experience in Go, Python, or similar languages.
- Proven experience establishing and maintaining an SLO program with operational rigor.
- Practical experience in reliability practices such as chaos engineering and failure injection.
- Ability to influence across team boundaries through credibility and expertise.
- 10+ years of industry experience with a bachelor's or master's degree, or equivalent experience operating systems at scale.
Nice To Haves
- Experience in a world-class reliability function such as Google SRE or Meta production engineering.
- Expertise operating GPU, HPC, or AI training infrastructure and understanding its failure modes.
- A track record of measurable reliability improvements within an organization.
- Proficiency with modern observability and operational tools such as Prometheus, OpenTelemetry, Grafana, PagerDuty, and Rootly.
Responsibilities
- Build an organization-wide reliability strategy and guide the maturity of operational practices in a 24/7 environment.
- Establish and maintain a rigorous SLO program and high standards across teams.
- Lead responses to high-severity incidents, emphasizing clear communication and effective resolution.
- Build and improve production code, including the data platform and related tooling.
- Implement chaos engineering, failure injection, and resilience testing.
- Improve team standards through hands-on technical example and leadership.
Benefits
- Equity
- Comprehensive benefits package
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
