Technical Product Manager - AI Infra Resilience

NVIDIA•Santa Clara, CA
•$195,000 - $442,000•Hybrid

About The Position

NVIDIA is hiring a Senior Product Manager to own workload failure attribution for the NVIDIA DSX Platform, an open, modular infrastructure software platform for designing, operating, and optimizing AI factories at scale. The role focuses on enabling GPU clusters to determine whether a failed workload was caused by hardware, system software, or application code, and surfacing that verdict to operators, schedulers, and the open-source developer surface built on the platform. The successful candidate will own the roadmap and collaborate with engineering, operations, cloud partners, cluster operators, and the developer community. Prior experience holding a product management title is not required; relevant experience as a technical product manager, architect, SRE, or infrastructure engineer is welcome.

Requirements

  • 12+ years of total experience.
  • 5+ years in product management, solutions architecture, software engineering, or site reliability engineering on technical infrastructure products.
  • Technical depth in GPU or AI infrastructure.
  • Familiarity with GPU failure modes, including XID codes, correctable and uncorrectable ECC errors, DCGM metrics, and NVLink and PCIe health signals.
  • Technical understanding of AI workload failures, including GPU hardware failure modes, distributed training failure patterns such as NCCL timeouts and framework errors, distinguishing hardware, system software, and application faults, and telemetry systems such as DCGM, sysfs, and dmesg.
  • Demonstrated ability to work with senior technical customers and translate operational requirements into product decisions.
  • Strong written and verbal communication across technical and non-technical audiences.
  • Bachelor's degree in Computer Science or equivalent experience.

Nice To Haves

  • Former SRE or infrastructure engineer working on GPU clusters, with experience triaging job failures, parsing XID logs, and investigating failures in large-scale training runs.
  • Deep familiarity with GPU scheduling, topology-aware placement, or multi-tenant GPU cluster management, including Slurm prolog/epilog, Kueue, or Kubernetes-native scheduler environments.
  • Experience building products that produce structured diagnostics or attribution output, such as health verdict schemas, log analysis pipelines, rules-engine systems, or observability APIs at data center scale.
  • Practical experience delivering or contributing to an open-source infrastructure project, including managing contributor engagement on GitHub.

Responsibilities

  • Drive the product vision, roadmap, and delivery for workload failure attribution in collaboration with engineering on architecture and prioritization.
  • Identify customer and partner gaps in GPU cluster failure triage at scale and define attribution capabilities to address them, including classification of NCCL timeouts and decisions about which signals belong in a public API versus an entitled runtime extension.
  • Work with cloud partners and cluster operators to translate operational triage requirements into platform capabilities.
  • Drive the open-source attribution project's community engagement strategy, including contribution models, ecosystem partnerships, and developer adoption.

Benefits

  • Equity
  • Benefits
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service