About The Position

Runware is hiring a senior staff-level individual contributor to own the GPU software stack across its infrastructure nodes, from firmware, kernel and NVIDIA driver through CUDA, container runtime, Kubernetes integration and GPU access for inference services. The role combines deep systems debugging with daily software development and works closely with node operations, Managed Kubernetes, inference performance and Sonic Inference Engine teams. The engineer will establish and maintain tested, reproducible GPU software baselines; qualify driver and CUDA releases across GPU generations, frameworks and model families; diagnose GPU failures across hardware and software layers; and build acceptance, diagnostics, telemetry and automated remediation systems. The role also covers GPU workload exposure and isolation on bare metal and Kubernetes, bringing up new GPU generations, benchmarking real inference workloads, working with NVIDIA and upstream projects, and using AI tools to increase engineering output. Success means GPUs run known and tested software combinations, fleet upgrades proceed without failed inferences, faults are detected and handled automatically, and new GPU generations can move from delivery to serving traffic in days. The position is described as remote-first, with twice-yearly in-person company gatherings, core collaboration hours and flexible scheduling outside those hours.

Requirements

  • 12 or more years building and operating systems software, including several years working with GPU software running in production at scale.
  • Deep, hands-on knowledge of the NVIDIA software stack, including kernel drivers and modules, CUDA runtime and libraries, NVML, DCGM and Container Toolkit, and how versions interact.
  • Demonstrated ability to find root causes of GPU failures across hardware, firmware, driver, runtime and application layers, including Xid errors, ECC and row remapping, PCIe and NVLink errors, and thermal and memory faults.
  • Strong Linux internals knowledge, including the kernel, device drivers and modules, PCIe, NUMA, IOMMU, cgroups and containers, and diagnostic tools such as perf, eBPF and ftrace.
  • Fluency in Python and at least one systems language (C, C++, Rust or Go), with the ability to write code regularly and use CUDA to create tests or reproduce bugs.
  • Experience operating GPUs under Kubernetes and on bare metal, including device plugins, GPU Operator, MIG, image builds and provisioning at scale.
  • Working knowledge of GPU use in inference frameworks such as PyTorch, TensorRT and ONNX Runtime, including memory allocation, compilation, caches and multi-GPU execution.
  • Clear technical writing, including design documents, postmortems and actionable ticket updates.
  • Regular use of AI to accelerate personal output and a clear perspective on how a small team should work with these tools.
  • Ability to own outcomes in a fast-moving, ambiguous environment, prioritize important problems and be accountable for results.

Nice To Haves

  • Experience building GPU drivers, CUDA libraries or GPU system software at a GPU vendor, cloud provider or HPC centre.
  • Experience with Blackwell-generation hardware (B200, GB200/GB300 NVL72), NVLink and NVSwitch topologies, and liquid-cooled racks.
  • Experience with multi-GPU and multi-node execution, including NCCL, GPUDirect and RDMA over RoCE or InfiniBand.
  • Firmware qualification across BIOS, BMC, NIC and GPU, and acceptance testing with OEMs and ODMs.
  • Experience with inference serving internals, including PyTorch, TensorRT-LLM, vLLM, SGLang or custom diffusion and video pipelines, and how they load, schedule and fail.
  • Experience with non-NVIDIA accelerators, such as AMD or custom silicon, and their software stacks.
  • An open-source project the candidate wrote or maintains that is used in production.
  • Experience with multi-tenant isolation and confidential computing on shared GPU hardware.
  • Experience operating a 24x7 service with published service levels.

Responsibilities

  • Own versioned, reproducible GPU software baselines spanning kernel, NVIDIA driver, firmware, CUDA runtime and libraries, and container toolkit; test changes before rollout and deploy across the fleet without downtime.
  • Qualify driver and CUDA releases across GPU generations, frameworks and model families, maintaining a compatibility matrix and catching correctness, performance and memory regressions before production.
  • Investigate GPU failures—including Xid errors, CUDA errors, cuBLAS failures, out-of-memory and fragmentation, and GPUs falling off the bus—and determine whether the cause is hardware, firmware, driver, runtime or application software.
  • Build GPU acceptance and diagnostic systems, including burn-in, DCGM diagnostics, memory and interconnect tests, real-workload tests, node and rack gating, and customer handover reports.
  • Combine telemetry from DCGM, the driver, kernel logs and BMC into a trusted GPU health view; classify faults and automate drain, reset and replacement workflows.
  • Own how GPUs are exposed to workloads, including the container runtime, device plugins, GPU Operator, MIG and sharing policies, and tenant isolation on bare metal and Kubernetes.
  • Bring up new GPU generations and pods with hardware and node operations teams, including initial driver and firmware setup, NVLink and NVSwitch validation, and bring-up automation.
  • Benchmark image, video, audio and LLM inference across GPU generations and software stacks, and use results to inform placement, pricing and purchasing.
  • Work with NVIDIA and upstream projects to report reproducible driver and firmware bugs, follow issues through resolution, track release changes and contribute fixes to open-source tools.
  • Use AI agents and automation for log and Xid triage, test generation, release qualification and runbooks.

Benefits

  • Generous paid time off, including vacation, sick days and public holidays.
  • Meaningful stock options.
  • Flexible hours outside core collaboration blocks.
  • Paid maternity, paternity and caregiver leave.
  • Twice-yearly company retreats.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service