Technical Product Manager - AI Infra Resilience

NVIDIA•Santa Clara, CA

21h•$208,000 - $379,500•Hybrid

About The Position

NVIDIA is hiring a Senior Product Manager to own workload failure attribution within its DSX Platform, an open, modular infrastructure software platform for designing, operating, and optimizing AI factories at scale. The role focuses on helping GPU clusters determine whether a failed workload was caused by hardware, system software, or application code, and surfacing that verdict to operators, schedulers, and the open-source developer platform built on top of it. The Product Manager will own the roadmap and product delivery, collaborating closely with engineering on architecture and prioritization and working with engineering and operations communities that deploy and extend the platform. The role will identify gaps in customer and partner failure-triage practices, define capabilities and telemetry/API boundaries, and engage cloud partners and cluster operators to translate operational needs into platform features. It will also drive community strategy for the open-source attribution project, including contribution models, ecosystem partnerships, and developer adoption. A prior product-management title is not required for candidates with deep technical experience in GPU cluster failure triage.

Requirements

12+ years of total experience.
5+ years in product management, solutions architecture, software engineering, or site reliability engineering on technical infrastructure products.
Technical depth in GPU or AI infrastructure.
Familiarity with GPU failure modes, including XID codes, correctable and uncorrectable ECC errors, DCGM metrics, and NVLink and PCIe health signals.
Technical understanding of AI workload failures, including GPU hardware failure modes, distributed training failure patterns such as NCCL timeouts and framework errors, distinguishing hardware, system software, and application faults, and telemetry systems such as DCGM, sysfs, and dmesg.
Demonstrated ability to work with senior technical customers and translate operational requirements into product decisions.
Strong written and verbal communication across technical and non-technical audiences.
Bachelor's degree in Computer Science or equivalent experience.

Nice To Haves

Former SRE or infrastructure engineer working on GPU clusters, with experience triaging job failures, parsing XID logs, and participating in cross-team investigations of large training-run failures.
Deep familiarity with GPU scheduling, topology-aware placement, or multi-tenant GPU cluster management, including Slurm prolog/epilog, Kueue, or Kubernetes-native scheduler environments.
Experience building products that produce structured diagnostics or attribution output, such as health verdict schemas, log analysis pipelines, rules-engine systems, or observability APIs at data-center scale.
Practical experience delivering or contributing to an open-source infrastructure project, including managing contributor engagement on GitHub.

Responsibilities

Drive the product vision, roadmap, and delivery for workload failure attribution, collaborating with engineering on architecture and prioritization.
Identify gaps in how customers and partners triage GPU cluster failures at scale and define attribution capabilities to address them.
Determine how failures such as NCCL timeouts are attributed to hardware, system software, or application causes, and which signals belong in a public API versus an entitled runtime extension.
Work directly with cloud partners and cluster operators to translate operational triage requirements into platform capabilities.
Drive community engagement strategy for the open-source attribution project, including contribution models, ecosystem partnerships, and developer adoption.