GPU Infrastructure Engineer

Rune•Mountain View, CA

2h•Onsite

About The Position

Rune connects stranded solar and wind power to computing demand. It deploys modular micro data centers directly at solar sites, using power that might otherwise be curtailed or unavailable to the grid. Rune is backed by Spark Capital, Union Square Ventures, and Lowercarbon Capital. The GPU Infrastructure Engineer is a hands-on, customer-facing engineer responsible for technical support of GPU clusters, including networking, node provisioning, and GPU health. This engineer serves as the primary responder for customer cluster issues, independently diagnosing and resolving problems or routing them appropriately while communicating clearly about confirmed findings and hypotheses. The role is based in Mountain View, California, and is not open to remote work. Rune offers competitive base compensation and strategic ownership in its infrastructure build-out. The stated compensation range is $175,000–$260,000.

Requirements

Hands-on experience with RDMA fabrics, specifically RoCEv2, including ECN/DCQCN, PFC, and DCBX trust-mode configuration on Mellanox/NVIDIA ConnectX NICs.
Experience debugging multi-node GPU interconnect performance using NCCL and RDMA benchmarking tools such as perftest and nccl-tests.
Working knowledge of GPU interconnect topology, including NVLink/NVSwitch and NUMA/PCIe affinity, and its effect on distributed training performance.
Experience reading GPU health telemetry using NVIDIA DCGM, nvidia-smi, and Xid codes, and managing driver/firmware compatibility across a fleet.
Strong Linux systems administration skills, including SSH/host security tooling such as fail2ban and sshd, and fleet automation using Ansible or an equivalent.
Ability to own live, customer-facing escalations on production systems and communicate clearly in writing and verbally under time pressure.

Nice To Haves

Familiarity with native InfiniBand fabrics, including OpenSM/UFM, in addition to RoCEv2.
Experience with scheduler-level network integration such as Slurm, Kubernetes, or Run:AI, including SR-IOV and multi-NIC bonding.
Prior experience in a vendor-facing escalation role, such as NVIDIA/Mellanox support, or as a Field Applications Engineer.

Responsibilities

Serve as the primary technical responder for customer-reported networking, GPU-interconnect, and node issues on live clusters.
Diagnose multi-node NCCL and RDMA performance regressions, including GPUDirect RDMA contention, and distinguish confirmed findings from working hypotheses when communicating with customers.
Configure and validate RoCEv2 congestion control, including ECN/DCQCN, PFC, and DCBX, and reconcile host-side NIC settings with switch-fabric behavior.
Diagnose GPU hardware faults using DCGM telemetry and Xid error codes, distinguishing hardware failures requiring RMA from driver, firmware, or configuration issues.
Own driver and firmware compatibility across the fleet, covering GPU drivers, NIC firmware (OFED/MOFED), and the kernel; validate compatibility before and after updates.
Provision, re-provision, and decommission GPU nodes, including power-cycling, secure wiping, re-imaging, BIOS/NUMA/PCIe topology validation, and acceptance testing before customer handoff.
Triage edge-network and jumpbox access issues affecting customer provisioning workflows, including automated blocking of legitimate high-concurrency SSH/Ansible traffic.
Write incident updates and post-resolution summaries for customer engineering teams, and feed recurring issues back into provisioning defaults and acceptance-test procedures.