Senior Resiliency Architect, Hardware Diagnostics and Resiliency Modeling

NVIDIA•Santa Clara, CA

3d•$184,000 - $287,500

About The Position

NVIDIA is seeking a Senior Resiliency Architect to support development of hardware diagnostics and resiliency models for Datacenter and Edge Computing products, including applications in Agentic and Physical AI. The role contributes to hardware diagnostics and resiliency for GPUs and SoCs used across product areas such as artificial intelligence, self-driving cars, and robotics. The architect will develop diagnostics and reliability models, analyze hardware failures and field data, improve fault attribution and recovery, and create automation and dashboards for diagnostic jobs across compute clusters. The role involves working with engineering teams to ensure telemetry correctly categorizes failures and keeping diagnostics and availability models current as AI workloads change.

Requirements

Master’s degree in Computer Engineering, Electrical Engineering, Computer Science, or a closely related field, or equivalent experience.
At least 5 years of relevant experience.
Knowledge of GPU architecture and programming, and hardware diagnostics.
Good understanding of reliability (FIT) models, AVF estimation, availability concepts, and ECC, parity, and CRC strategies.
Proficiency in data mining, automation, AI tool usage, and scripting with Python, SQL, or similar.
Proficiency in C/C++, CUDA programming, and use of AI tools for coding and debugging.
Strong debugging and analytical skills.
Excellent interpersonal skills and ability to collaborate with on-site and remote teams.
Working knowledge of hardware diagnostics and container-based job execution.

Nice To Haves

Strong statistical judgment, particularly around exposure-adjusted rates, sample size, and distinguishing signal from artefact.
Strong understanding of resiliency and reliability trade-offs in AI datacenters, including failure modes, mitigation and recovery strategies, and impacts on large-scale training and inference workloads.
Experience developing embedded software or visualization and data analytics tools.

Responsibilities

Develop and maintain a diagnostics software suite that stress tests NVIDIA GPUs and SoCs to identify hardware defects, including defects that cause silent data corruption.
Run diagnostics in large-scale datacenter GPU and Safety SoC deployments across package, board, and rack configurations involving GPUs, CPUs, and Networking SoCs.
Analyze coverage gaps in the diagnostic suite identified through silicon failures on customer workloads or test suites, and improve coverage, failure repeatability, and test time.
Model and analyze reliability metrics, including FIT for permanent and transient errors and availability from GPU through rack and datacenter levels.
Use reliability models to identify gaps and drive improvements through new hardware and software features.
Analyze field data on hardware interruptions and permanent failures, and refine RAS models to better correlate with field data.
Ensure optimal fault attribution, containment, and recovery for hardware errors.
Design, build, and maintain automation scripts and tools to launch, monitor, and manage diagnostic jobs across multiple independent compute clusters.
Develop and maintain dashboards that integrate diverse data sources and formats, provide analytics and visualization, and support reliable hosting, secure access, and multi-user performance.
Collaborate with engineering teams to ensure system telemetry accurately categorizes cluster failures as hardware faults, test-script bugs, or environment configuration issues.
Ensure availability models and hardware diagnostics evolve to reflect changes in AI workloads.