Senior Software Engineer, AI Compute Scheduling

NVIDIA•Durham, NC

3d•$184,000 - $356,500

About The Position

NVIDIA’s Managed AI Research Superclusters (MARS) team builds infrastructure, platforms, and tools for AI/ML research and engineering. The MARS Scheduling team determines how NVIDIA’s AI research compute is used across large GPU clusters spanning multiple clouds and partners. Its systems support workloads including frontier-model pretraining, reinforcement learning, and agentic workloads, using Slurm and Kubernetes at large scale. The Senior Software Engineer will design, build, and operate scheduling systems for large GPU fleets. The work includes topology-sensitive scheduling on rack-scale systems, policy enforcement for research-team allocations, cloud-native Slurm services, Kubernetes batch scheduling, and services for reservations, node health, workflows, and APIs. The engineer will measure and improve GPU utilization, fairness, waste, and job start times, own the reliability of components they build, and collaborate with internal teams and open-source communities.

Requirements

BS or MS in Computer Science, Computer Engineering, Electrical Engineering, or a related field, or equivalent experience.
At least 6 years of experience building production systems software.
Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn, or a scheduler you built, including understanding its internals.
Strong programming skills in Go, C, or C++, plus Python for tooling and analysis.
Hands-on Kubernetes experience with controllers or operators, CRDs, scheduling, and resource management.
Solid Linux systems knowledge, including containers (Docker, Podman, Enroot, or Apptainer), cgroups, networking basics, and debugging under load.
A track record of owning a production service and measurably improving it.
Clear written and spoken communication with engineers and researchers.

Nice To Haves

Experience scheduling large-scale AI training involving thousands of GPUs, NVLink and InfiniBand or RoCE topology, and multi-node jobs that must start together.
Contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI, or similar open-source schedulers.
Experience running Slurm on Kubernetes (Slinky or similar), or building Kubernetes operators for batch or HPC workloads.
Familiarity with reinforcement learning, agentic, or inference workloads and their scheduling needs, including heterogeneous jobs, sandboxes, elastic scale, and fast restarts.
Experience with workflow orchestrators such as Flyte, Airflow, Ray, or Kubeflow at scale.

Responsibilities

Design and build scheduling for large GPU fleets using Slurm and Kubernetes, including queueing, gang and topology-aware placement on rack-scale NVLink systems, preemption, reservations, and backfill.
Build a policy layer for fair share, quotas, and allocations across research teams, driven by a central allocation system and consistently enforced across schedulers.
Run Slurm as a cloud-native service on Kubernetes and extend Kubernetes batch scheduling, including Kueue and its ecosystem, for training, reinforcement learning, sandboxes, and inference.
Build scheduler-related services for reservation automation, node-health-aware placement, workflow orchestration across clusters, and APIs for submitting and tracking work.
Measure and improve goodput, including GPU occupancy, waste, fairness, and time to start for large jobs.
Own reliability for built components through on-call, incident analysis, and fixes that prevent recurring classes of failure.
Contribute to Slurm, Kubernetes, and related open-source projects, and partner with platform, storage, networking, and research teams.