GPU Infrastructure Engineer

Rune•Mountain View, CA

3h•$175,000 - $260,000•Onsite

About The Position

Rune is hiring a hands-on GPU Infrastructure Engineer to own customer-facing technical support for GPU clusters, covering networking, node provisioning, and GPU health. The engineer will be the primary responder responsible for ensuring customer clusters perform as expected by diagnosing and resolving issues or routing them appropriately while retaining technical ownership. The role is based in Mountain View, California. Rune builds modular micro data centers at solar sites to convert otherwise unused power into compute, and is backed by Spark Capital, Union Square Ventures, and Lowercarbon Capital.

Requirements

Hands-on experience with RDMA fabrics, specifically RoCEv2, including ECN/DCQCN, PFC, and DCBX trust-mode configuration on Mellanox/NVIDIA ConnectX NICs.
Experience debugging multi-node GPU interconnect performance with NCCL and RDMA benchmarking tools such as perftest and nccl-tests.
Working knowledge of GPU interconnect topology, including NVLink/NVSwitch and NUMA/PCIe affinity, and its effects on distributed training performance.
Experience reading GPU health telemetry using NVIDIA DCGM, nvidia-smi, and Xid codes, and managing driver/firmware compatibility across a fleet.
Strong Linux systems administration skills, including SSH/host security tooling such as fail2ban and sshd, and fleet automation using Ansible or equivalent.
Comfort owning live, customer-facing escalations on production systems and communicating clearly in writing and verbally under time pressure.

Nice To Haves

Familiarity with native InfiniBand fabrics, including OpenSM/UFM, in addition to RoCEv2.
Experience with scheduler-level network integration such as Slurm, Kubernetes, or Run:AI, including SR-IOV and multi-NIC bonding.
Prior experience in a vendor-facing escalation role, such as NVIDIA/Mellanox support, or as a Field Applications Engineer.

Responsibilities

Serve as the primary technical responder for customer-reported networking, GPU-interconnect, and node issues on live clusters.
Diagnose multi-node NCCL and RDMA performance regressions, including issues related to GPUDirect RDMA contention, and distinguish confirmed findings from working hypotheses when communicating with customers.
Configure and validate RoCEv2 congestion control, including ECN/DCQCN, PFC, and DCBX, and reconcile host-side NIC settings with actual switch-fabric behavior.
Diagnose GPU hardware faults using DCGM telemetry and Xid error codes, distinguishing hardware failures requiring RMA from driver, firmware, or configuration issues.
Own driver and firmware compatibility across the fleet, including GPU drivers, NIC firmware (OFED/MOFED), and kernel versions, and validate compatibility before and after updates.
Provision, re-provision, and decommission GPU nodes, including power-cycling, secure wiping, re-imaging, BIOS/NUMA/PCIe topology validation, and acceptance testing before customer handoff.
Triage edge-network and jumpbox access issues that affect customer provisioning workflows, such as legitimate high-concurrency SSH/Ansible traffic being automatically blocked.
Write incident updates and post-resolution summaries for customer engineering teams, and feed recurring issues back into provisioning defaults and acceptance-test procedures.