Senior Validation Technical Program Manager, Logical Deployment

Nscale • Seattle, WA

2d

About The Position

The Senior Validation Technical Program Manager, Logical Deployment, owns proof that AI infrastructure clusters are ready for customer handoff. The role translates customer acceptance criteria into agreed test plans, runs cluster-scale validation, tracks results through signed acceptance, and produces evidence-based reports for customers and leadership. Validation covers performance, interconnect, storage, stability, and burn-in.

This is an individual-contributor role between engineering, OEM partners, and customers. It requires hands-on software validation and test-framework experience in addition to program ownership: reviewing raw outputs, investigating unexpected results, writing or modifying scripts to unblock tests, triaging failures with evidence, and verifying fixes before marking tests passed. The role also improves reusable validation playbooks, test suites, and reporting automation. It reports to the Director, Logical Deployment and involves travel of up to 50%, primarily to data center sites.

Requirements

• 7–10+ years in software validation, test engineering, or QA for infrastructure or distributed systems, including program or project ownership.
• Hands-on experience building and running test frameworks and reading and debugging their output.
• Strong scripting skills in Python and Bash, and ability to modify test harnesses and data pipelines.
• Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics, and burn-in.
• Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.
• Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.
• Clear, concise communication with engineers and customers, both in writing and in meetings.
• Ability to travel up to 50% of the time, primarily to data center sites.
• Rigorous, evidence-based approach to validation and willingness to investigate anomalous results.
• Ability to make and communicate readiness decisions, including calling a cluster not ready when results warrant it.
• Ability to hold engineering, OEM partners, and vendors accountable for fixing test findings and to escalate early with evidence.
• Accountability for validation outcomes and initiative in unblocking work.
• Ability to keep testing moving under pressure and near handoff deadlines.
• Technical ability to read and interpret test logs, performance traces, and scripts.

Nice To Haves

• Experience validating NVIDIA GB200/GB300 or Vera Rubin NVL72-class clusters, including NVIDIA cluster validation tooling.
• Experience with MLPerf or customer-defined training and inference benchmarks at scale.
• Familiarity with Slurm or Kubernetes for scheduling large test jobs.
• Experience at a hyperscaler, neocloud, or AI lab accepting 10K+ GPU clusters.
• Background in CI/CD and infrastructure test automation.

Responsibilities

• Translate customer acceptance criteria into clear test plans with test scope, scale, and pass thresholds.
• Agree validation plans with customers and engineering before testing begins.
• Map each acceptance criterion to evidence from test results and maintain traceability.
• Drive end-to-end cluster acceptance validation, including performance, interconnect, storage, stability, and burn-in.
• Track validation status for every cluster, including passed, failed, and blocked tests.
• Coordinate test windows with fabric, provisioning, and OEM teams.
• Review raw test output, investigate unexpected results, and identify root causes.
• Write or modify scripts when tests, harnesses, or data collection steps are missing or broken.
• Triage failures to the appropriate hardware, fabric, firmware, software, or configuration owner, provide evidence, and drive issues to closure.
• Rerun tests and verify fixes before marking results as passed.
• Produce acceptance reports that connect results to criteria, explain exceptions, and support customer sign-off.
• Brief customers and leadership on validation status and risks to handoff dates.
• Improve validation playbooks, test suites, and automation based on lessons learned.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed