Senior Site Reliability Engineer - HPC

NVIDIA•Durham, NC

4d•$152,000 - $287,500•Hybrid

About The Position

NVIDIA is looking for a Senior SRE to join their Compute Farm team to help build the next generation of their global services platform. In this role, you will be responsible for keeping critically important systems running while working on technologies that redefine computing. You will harness the power of AI to deliver groundbreaking solutions and see your work have a lasting impact.

Requirements

B.S. degree in Computer Science or related technical field (or equivalent experience) with 5+ years professional experience building and supporting critical services.
Experience supporting large‑scale HPC clusters using Slurm, LSF or Kubernetes clusters, including setup, tuning, and troubleshooting.
Proficiency in modern CI/CD techniques, and Infrastructure as Code (IaC) for managing services.
Strong experience crafting large-scale infrastructure platforms for automated host lifecycle management, fleet reliability/auto-healing, E2E observability or data-driven operations (AIOps/ML-driven signals) that materially reduce manual intervention.
Proficient in monitoring, metrics, container management, and log collection tools.
5+ years of coding/scripting experience in at least two high‑level programming languages such as Python, Go, Perl, or Ruby.
Mentored other engineers and influenced technical direction through design reviews, architecture documents, and strong partnership with product and leadership.
Creative problem solver with excellent debugging skills and strong communication and documentation abilities.

Nice To Haves

Published technical write‑ups or talks (conference presentations, meetups, engineering blogs) that deep‑dive into real‑world reliability, observability, or large‑scale HPC/SRE problems and their solutions.
Maintainer or co‑maintainer responsibilities for an open source component used in production (plugins, operators, exporters, controllers, or SDKs) at large scale.

Responsibilities

Own SRE solutions end‑to‑end, from design and implementation to operation and continuous improvement, ensuring they integrate cleanly with HPC schedulers, storage, and network fabrics.
Use IaC (Infrastructure‑as‑Code) and config management to standardize and automate provisioning everywhere.
Deliver solutions in a globally distributed, multi‑cloud hybrid environment – On‑prem, AWS, GCP, and OCI.
Design for failure with redundancy, failure domains, progressive delivery, and strict change control.
Ensure the highest level of uptime and Quality of Service (QoS) for internal customers through operational excellence.
Conduct capacity management and planning to meet ongoing operational needs.
Detect performance issues and recommend solutions to maintain world‑class service quality.
Collaborate with various teams in a fast‑paced environment to ensure seamless project completion.
Participate in on-call, incident reviews, assist in root cause identification, and produce high-quality RCA reports.