Principal Engineer, Inference Memory and Storage Systems

DigitalOcean • Seattle, WA

13d • Hybrid

About The Position

DigitalOcean is seeking a Principal Engineer to lead the technical vision and roadmap for the memory and storage layer of its inference platform. This role is crucial for optimizing the performance and cost-effectiveness of running large language models (LLMs) on DigitalOcean's GPU infrastructure. The Inference Platform team is responsible for the serving stack that handles frontier open models at production scale, including model orchestration, disaggregated prefill and decode, request routing, and engine integrations. As traffic patterns evolve and context lengths increase, efficient KV cache management becomes paramount. This position will unify existing solutions into a cohesive, multi-tier memory and storage substrate, spanning GPU HBM, host DRAM, local NVMe, and remote object storage, accessible through a standardized interface to all serving engines and routers. The selected candidate will guide cross-team technical direction, contribute to open-source projects, and translate complex systems work into tangible unit economics for customers.

Requirements

• 15+ years of experience building large-scale distributed systems, high-performance storage, or ML systems infrastructure, with a proven track record of delivering and operating production services
• Deep fluency in memory hierarchies (GPU HBM, host DRAM, NVMe, remote/object storage) and experience designing systems that span tiers for performance and cost
• Experience with distributed caching or key-value systems, particularly designs optimized for low latency under high concurrency
• Hands-on experience with networked I/O and RDMA/NVMe-oF/NVLink-class technologies
• Familiarity with aggregated and disaggregated deployment topologies for AI clusters
• Strong systems programming skills in C/C++, Go, Rust, or Python
• Comfort reading and modifying serving-engine internals
• Rigor in profiling and optimization across CPU, GPU, memory, and network, using measurement to drive architectural decisions and validate improvements in TTFT, ITL, and throughput
• Excellent written and verbal communication skills
• History of leading cross-functional efforts with product, infrastructure, and customer-facing teams

Nice To Haves

• Contributions to open-source LLM serving or inference infrastructure projects (e.g., vLLM, SGLang, llm-d, NVIDIA Dynamo, LMCache), especially related to KV cache offload, compression, or reuse
• Experience designing a unified memory or storage layer that presents a single logical KV or object model across GPU, host, SSD, and cloud tiers in a hyperscale or public cloud environment
• Familiarity with Kubernetes-based GPU orchestration, including DRA, MIG/MPS partitioning, and gateway/inference-extension routing patterns
• Publications or patents in LLM systems, memory-disaggregated architectures, RDMA-based data planes, or CDN-like caching systems for ML workloads

Responsibilities

• Defining and evolving a unified memory layer spanning GPU memory, pinned host memory, RDMA-accessible memory, local NVMe tiers, and remote object storage for large-scale LLM inference
• Architecting deep integrations with the serving engines (vLLM, SGLang, TensorRT-LLM), focusing on KV cache offload, reuse, eviction policy, and cross-node sharing
• Designing interfaces and protocols for disaggregated prefill/decode, peer-to-peer KV cache transfer, and cache-aware routing
• Owning the eviction and admission strategy across memory tiers, including LRU, cost-aware policies, and pluggable backends, and defining metrics to validate policy effectiveness
• Partnering with GPU infrastructure, networking, and platform teams to leverage technologies like GPUDirect, RDMA, NVMe-oF, and NVLink for low-latency cache access across heterogeneous accelerator pools
• Driving unit economics by modeling and validating the impact of cache hit ratio, prefix caching, and tiering on cost metrics, and communicating these tradeoffs to product and pricing teams
• Setting technical direction, raising standards through design reviews, mentoring senior and staff engineers, and sponsoring roadmap initiatives
• Representing DigitalOcean externally through upstream contributions, conference presentations, and customer-facing technical discussions

Benefits

• Competitive array of benefits to support you from our Employee Assistance Program to Local Employee Meetups to flexible time off policy
• Reimbursement for relevant conferences, training, and education
• Access to LinkedIn Learning's 10,000+ courses
• Bonus in addition to base salary
• Equity compensation to eligible employees, including equity grants upon hire
• Option to participate in our Employee Stock Purchase Program

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed