Senior Systems Software Engineer, Workload Systems Engineering

NVIDIA•Santa Clara, CA

1d•$184,000 - $356,500

About The Position

NVIDIA is seeking a senior systems software engineer for its Workload Systems Engineering team. The team connects customer workflows with the infrastructure that performs demanding build, simulation, and validation workloads. It develops self-service mechanisms and clear diagnostics, investigates performance and reliability across system boundaries, and delivers measured infrastructure improvements with the teams that own underlying components. The role enables application configuration, remote execution, workers, hosts, GPU and CPU resources, storage, data paths, and on-target systems. It involves directly evaluating systems, diagnosing causes of slow or failing workloads, testing alternatives in representative scenarios, applying improvements with others, and verifying customer benefits.

Requirements

Over 8 years of solid software engineering experience working with production compute infrastructure, distributed systems, large-scale build and test systems, or comparable systems software.
Hands-on experience debugging complex performance or reliability issues across software and hardware boundaries using logs, metrics, traces, profiling, and experiments.
Proficiency in a systems programming or automation language and experience delivering tested changes into production infrastructure.
In-depth experience with Linux and proficiency in one or more of the following: GPU systems and drivers, scheduling and resource management, remote execution or CI infrastructure, storage and data paths, or host and container infrastructure.
Ability to work across teams with different component ownership, turn ambiguous symptoms into a clear technical plan, and explain results to engineering customers.
A degree in computer science, computer engineering, or a related field, or equivalent experience.

Nice To Haves

Experience with GPU sharing and isolation, CUDA execution behavior, MIG or MPS or equivalent experience, GPU driver diagnostics, or high-performance computing workloads.
Experience with workload placement, CPU and GPU concurrency, NUMA, cloud-instance benchmarking, or cost and performance tradeoffs.
Familiarity with Bazel, remote execution, build front ends, simulation or replay workloads, and hardware-in-the-loop validation.
Experience improving file-system, caching, networking, or high-throughput data paths under concurrent workloads.
A record of building customer self-service or observability tools and using incident and retry data to drive verified reliability improvements.

Responsibilities

Build shared self-service and validation mechanisms that let engineering teams configure and use supported infrastructure workflows with less manual assistance.
Provide customers and on-call teams with diagnostic views across application, submission, remote-execution, and host or target layers to help identify the source of failures and slowdowns.
Set performance and reliability benchmarks for representative workloads; use profiling, metrics, traces, and controlled experiments to identify effective changes.
Evaluate and improve compute and GPU selection, scheduling and placement, concurrency, worker and runner configuration, resource sharing, caching, and input/output paths.
Make retry causes, recovery outcomes, and wasted execution capacity visible, and work with component owners to eliminate recurring failures and unnecessary re-execution.
Implement and validate infrastructure-level changes with application, build and CI, remote-execution, and infrastructure teams; measure effects on turnaround, successful efficiency, stability, usable capacity, and cost.
Turn successful investigations into supported workflows, diagnostics, documentation, and clear customer handoffs.