Senior Validation Technical Program Manager, Logical Deployment

Nscale•San Francisco, CA

About The Position

The Senior Validation Technical Program Manager owns proof that AI infrastructure clusters are ready for customer acceptance. The role translates customer acceptance criteria into agreed test plans, runs validation at cluster scale, tracks results from the first test through signed acceptance, and produces evidence-based reports for customer handoff. It combines software validation and test-framework expertise with program ownership and requires hands-on investigation of test outputs and scripting when needed to unblock testing. As an individual contributor reporting to the Director, Logical Deployment, this role works across engineering, OEM partners, vendors, and customers. It owns validation across performance, interconnect, storage, stability, and burn-in; coordinates test windows; triages failures and verifies fixes; and communicates validation status and risks. The position also improves validation playbooks, test suites, and reporting automation. Travel of up to 50% is expected, primarily to data center sites.

Requirements

7–10+ years of experience in software validation, test engineering, or QA for infrastructure or distributed systems, including program or project ownership.
Hands-on experience building and running test frameworks and reading and debugging their output.
Strong scripting skills in Python and Bash, with the ability to modify test harnesses and data pipelines.
Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics, and burn-in.
Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency and storage throughput.
Experience translating requirements or acceptance criteria into test plans with clear pass/fail evidence.
Clear, concise written and verbal communication with engineers and customers.
Ability to travel up to 50% of the time, primarily to data center sites.

Nice To Haves

Experience validating NVIDIA GB200/GB300 or Vera Rubin NVL72-class clusters, including NVIDIA cluster validation tooling.
Experience with MLPerf or customer-defined training and inference benchmarks at scale.
Familiarity with Slurm or Kubernetes for scheduling large test jobs.
Experience at a hyperscaler, neocloud, or AI lab accepting 10K+ GPU clusters.
Background in CI/CD and test automation for infrastructure.

Responsibilities

Translate customer acceptance criteria into clear test plans, including test scope, scale, and pass thresholds.
Agree test plans with customers and engineering before testing begins.
Map each acceptance criterion to its validating test result and maintain traceability.
Drive end-to-end cluster acceptance validation, including performance, interconnect, storage, stability, and burn-in testing.
Track validation status for every cluster, including passed, failed, and blocked tests.
Coordinate test windows with fabric, provisioning, and OEM teams.
Review raw test output, investigate unexpected results, and identify root causes.
Write or modify scripts when tests, harnesses, or data-collection steps are missing or broken.
Triage failures to the appropriate hardware, fabric, firmware, software, or configuration owner, provide evidence, and drive issues to closure.
Re-run tests and verify fixes before marking results as passed.
Produce acceptance reports that link results to criteria, explain exceptions, and support customer sign-off.
Brief customers and leadership on validation status and risks to handoff dates.
Improve validation playbooks, test suites, and automation based on lessons learned.
Travel up to 50% of the time, primarily to data center sites.