Senior Resiliency Architect, Hardware Diagnostics and Resiliency Modeling

NVIDIA•Santa Clara, CA

About The Position

The Senior Resiliency Architect will support development of hardware diagnostics and resiliency models for Datacenter and Edge Computing products, including applications in Agentic and Physical AI. The role focuses on diagnostics and reliability for GPUs and SoCs used in products ranging from artificial intelligence systems to self-driving cars and robots. The position develops and maintains diagnostics software for testing NVIDIA GPUs and SoCs for hardware defects, including silent data corruption, in large-scale deployments and package, board, and rack configurations. It also analyzes diagnostic coverage and field data, models reliability and availability, and drives improvements to hardware and software features. The role includes building automation and dashboards for diagnostic operations and collaborating with engineering teams to accurately classify cluster failures. NVIDIA states that the expected base salary range is USD 184,000–287,500, with eligibility for equity and benefits. Applications will be accepted at least until November 7, 2026. This posting is for an existing vacancy.

Requirements

Master’s degree in Computer Engineering, Electrical Engineering, Computer Science, or a closely related field, or equivalent experience.
At least 5 years of relevant experience.
Knowledge of GPU architecture and programming and hardware diagnostics.
Good understanding of reliability (FIT) models, AVF estimation, availability concepts, and ECC, parity, and CRC strategies.
Proficiency in data mining, automation, AI tool usage, and scripting with Python, SQL, or similar.
Proficiency in C/C++, CUDA programming, and using AI tools for coding and debugging.
Strong debugging and analytical skills.
Excellent interpersonal skills and ability to collaborate with on-site and remote teams.
Working knowledge of hardware diagnostics and container-based job execution.

Nice To Haves

Strong statistical judgment, particularly concerning exposure-adjusted rates, sample size, and distinguishing signal from artifact.
Strong understanding of resiliency and reliability trade-offs in AI datacenters, including failure modes, mitigation and recovery strategies, and effects on large-scale training and inference workloads.
Experience developing embedded software or visualization and data analytics tools.

Responsibilities

Develop and maintain diagnostic software to stress test NVIDIA GPUs and SoCs and identify hardware defects, including defects that cause silent data corruption.
Run diagnostics in large-scale Datacenter GPU and Safety SoC deployments across package, board, and rack configurations spanning GPUs, CPUs, and Networking SoCs.
Analyze diagnostic coverage gaps identified by silicon failures on customer workloads or test suites.
Enhance diagnostics to improve coverage and repeatability of detected failures and optimize test time.
Model and analyze reliability, availability, and serviceability (RAS) metrics, including FIT for permanent and transient errors and availability from GPU through rack and datacenter levels.
Use models to identify gaps and drive RAS improvements through new hardware and software features.
Analyze field data on hardware interruptions and permanent failures and improve RAS models to better correlate with field data.
Ensure optimal fault attribution, containment, and recovery for hardware errors.
Design, build, and maintain automation scripts and tools to launch, monitor, and manage diagnostic jobs across multiple independent compute clusters.
Develop and maintain dashboards that integrate diverse data sources and formats, provide analytics and visualization, and support reliable hosting, secure access, and multi-user performance.
Collaborate with engineering teams to ensure system telemetry accurately classifies cluster failures as hardware faults, test-script bugs, or environment configuration issues.
Ensure availability models and hardware diagnostics evolve to reflect changes in AI workloads.