Principal Engineer, Rack-Scale GPU Architecture

DigitalOcean • Seattle, WA

5d • $249,600 - $312,000 • Hybrid

About The Position

DigitalOcean is seeking a Principal Engineer to lead the architecture, orchestration, and operation of rack-scale GPU systems for inference. This role is critical as the industry shifts from server-centric to rack-centric GPU capacity, with new hardware like GB300 NVL72 and future generations requiring a re-evaluation of scheduling, networking, and failure-handling models. The successful candidate will define the reference architecture for rack-scale inference, focusing on NVLink domain allocation, RDMA fabric design, Kubernetes topology awareness, and robust failure management. This position involves close collaboration with internal data center, networking, and inference platform teams, as well as direct engagement with NVIDIA for pre-silicon enablement and bringup.

Requirements

• 12+ years in large-scale compute, HPC, or GPU infrastructure engineering with hands-on production systems responsibility.
• Deep understanding of GPU interconnect and system topology (NVLink, NVSwitch, NVLink domains, PCIe, NUMA) and their impact on workload placement.
• Strong RDMA networking background (RoCEv2 or InfiniBand at scale, GPUDirect RDMA, congestion control, lossless fabric tuning, rail-optimized cluster topologies).
• Practical NCCL expertise, including debugging, algorithm/protocol selection, topology detection, and performance diagnostics.
• Substantial Kubernetes depth for accelerated workloads (device plugins, DRA, scheduling extensions, operators, understanding Kubernetes model limitations with rack-scale hardware).
• Strong systems programming and automation skills in Go and Python.
• Comfort operating at the driver, firmware, and BMC layer when necessary.
• Experience operating fleets with routine hardware failure, including diagnostics, automation, and blast-radius thinking.
• Excellent written and verbal communication skills.
• Track record of leading architecture across organizational boundaries (hardware, networking, platform, product).

Nice To Haves

• Direct experience bringing up GB200 or GB300 NVL72 systems, or comparable rack-scale platforms, in a production environment.
• Experience with multi-node inference for very large models (expert or pipeline parallelism spanning NVLink domains, and associated placement constraints).
• Familiarity with liquid-cooled and high-density rack deployments, and their operational realities.
• Contributions to relevant open source projects (Kubernetes SIG-Node/scheduling, NVIDIA GPU/Network Operator, DRA drivers, Kueue, LeaderWorkerSet, NCCL).
• Experience running heterogeneous fleets spanning multiple accelerator vendors and generations under a single orchestration model.

Responsibilities

• Owning the end-to-end reference architecture for rack-scale inference systems (GB300 NVL72, Vera Rubin-class NVL144, and successors), from fabric design to customer schedulability.
• Defining NVLink domain partitioning, allocation, and isolation across tenants and workloads, including multi-node NVLink and IMEX domain provisioning, fabric manager configuration, and blast radius assessment.
• Architecting the scale-out RDMA fabric (RoCEv2 or InfiniBand) with rail-optimized topologies, congestion control, and GPUDirect RDMA paths, tuning NCCL for optimal performance.
• Making Kubernetes topology-aware for these systems, including DRA for GPU/NVLink-domain allocation, multi-node inference primitives (LeaderWorkerSet, gang scheduling), Topology Manager, NUMA alignment, and underlying GPU/Network Operators.
• Driving multi-node disaggregated serving in partnership with the inference platform team, mapping prefill and decode pools to NVLink domains and rail boundaries, and managing KV cache movement.
• Owning failure semantics and serviceability at rack scale, including health checking, DCGM-based diagnostics, drain/repair workflows, degraded-domain scheduling, and fleet-wide firmware/driver lifecycle management.
• Establishing validation and burn-in methodologies for production qualification, including collective bandwidth/latency characterization, NCCL/inference benchmarks, and acceptance gates.
• Partnering with data center engineering on power, liquid cooling, and rack density constraints to ensure deployability and cost-effectiveness.
• Working directly with NVIDIA and ODM partners on pre-silicon enablement, early hardware bringup, and roadmap feedback.
• Setting technical direction, mentoring senior and staff engineers, and representing DigitalOcean upstream, at conferences, and in customer architecture reviews.

Benefits

• Competitive array of benefits to support you from our Employee Assistance Program to Local Employee Meetups to flexible time off policy.
• Reimbursement for relevant conferences, training, and education.
• Access to LinkedIn Learning's 10,000+ courses.
• Bonus in addition to base salary (based on company and individual performance).
• Equity compensation, including equity grants upon hire and the option to participate in our Employee Stock Purchase Program.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed