DSX NVCRE Senior Engineer

NVIDIA•Santa Clara, CA

16h

About The Position

NVIDIA is seeking a Senior Software Engineer to join its DSX team and build and operate tooling that verifies AI infrastructure is ready for production. The role focuses on cluster readiness and certification software, including the open-source Cluster Readiness Engine, which validates GPU clusters across Kubernetes and Slurm, measures training and inference performance, and produces trusted evidence for customers and partners. The engineer will design and implement cluster readiness and certification tools in Go and Python, including controllers, operators, command-line tools, and scheduler integrations. The work includes developing benchmarks and real workloads for communication, compute, storage, diagnostics, training, inference, reliability, and security, with architecture-specific expectations and clear pass/fail criteria. The role also develops provenance and evidence pipelines that record what ran, where it ran, and its configuration, and produce signed, auditable results. The engineer will integrate new GPU platforms, schedulers, and fabrics into testing, from single-node bring-up through multi-node production-cluster validation. They will collaborate with internal platform, networking, and solutions teams and cloud partners to investigate performance and correctness gaps, contribute to open-source projects, and author design documents, runbooks, documentation, and operational guides. NVIDIA describes itself as a leader in Artificial Intelligence, High-Performance Computing, and Visualization, and says its work enables advances from AI to autonomous cars. The position is an existing vacancy. Applications will be accepted at least until November 1, 2026. NVIDIA uses AI tools in recruiting and is an equal opportunity employer.

Requirements

Bachelor's, master's, or Ph.D. degree in Computer Science, Computer or Electrical Engineering, or a related technical field, or equivalent experience.
At least 8 years of software engineering experience building infrastructure or systems software, ideally for large-scale compute, cloud, or HPC environments.
Strong programming skills in Go and Python, with experience shipping production-quality services, controllers, or command-line tools.
Deep familiarity with Linux systems, containers, and Kubernetes, including writing operators or controllers against the Kubernetes API.
Working knowledge of HPC schedulers such as Slurm and GPU cluster fundamentals, including drivers, NCCL, InfiniBand or RoCE fabrics, and GPU health tooling such as DCGM.
Experience benchmarking or validating distributed systems, with a rigorous approach to measurement, reproducibility, and evidence.
Ability to lead complex technical investigations to root cause and communicate findings clearly to engineering and leadership audiences.

Nice To Haves

Hands-on experience running or tuning large-scale distributed training or inference workloads with frameworks such as PyTorch, Megatron, NeMo, vLLM, SGLang, or TensorRT-LLM.
Experience with the NVIDIA ecosystem, including DGX systems, GB200/GB300 platforms, CUDA, NVLink and NVSwitch fabrics, and NVIDIA networking.
Prior contributions to open-source infrastructure projects, especially Kubernetes operators, scheduler plugins, or benchmarking frameworks.
Background in software supply-chain security, attestation, or signed artifacts, and building audit-grade evidence into systems.
Familiarity with observability stacks such as Prometheus, Grafana, and OpenTelemetry, as well as GitOps workflows and infrastructure as code for GPU clusters.

Responsibilities

Design and build systems that make cluster validation a repeatable, verifiable software process.
Design and implement cluster readiness and certification tooling in Go and Python for Kubernetes and Slurm, including controllers, operators, command-line tools, and scheduler integrations.
Develop a catalog of benchmarks and real workloads covering communication, compute, storage, diagnostics, training, inference, reliability, and security, with architecture-specific expectations and pass/fail criteria.
Develop provenance and evidence pipelines that capture what ran, where it ran, and under which configuration, and produce signed, auditable results.
Integrate new GPU platforms, schedulers, and fabrics into testing, from single-node bring-up through multi-node validation on production clusters.
Collaborate with internal platform, networking, and solutions teams and cloud partners to investigate performance and correctness gaps and feed fixes back into platforms and reference architectures.
Contribute to the open-source Cluster Readiness Engine and related projects, including design documents, documentation, and reviews.
Author design documents, runbooks, and operational guides explaining cluster validation and how to interpret results.