Principal Software Engineer – Large-Scale LLM Memory and Storage Systems
About The Position
NVIDIA is seeking a Principal Systems Engineer to define the vision and roadmap for memory management in large-scale LLM and storage systems. The role focuses on evolving a unified memory layer across GPU memory, pinned host memory, RDMA-accessible memory, SSDs, and remote file, object, and cloud storage to support large-scale LLM inference. The engineer will build integrations with LLM serving engines such as vLLM, SGLang, and TensorRT-LLM, focusing on KV-cache offload, reuse, and remote sharing across heterogeneous and disaggregated clusters. The role also involves designing interfaces and protocols for disaggregated prefill, peer-to-peer KV-cache sharing, and multi-tier KV-cache storage. The engineer will work with GPU architecture, networking, and platform teams to use technologies such as GPUDirect, RDMA, and NVLink, and will provide technical direction, mentorship, and representation in reviews, open-source communities, conferences, and customer-facing technical discussions.
Requirements
- Master's degree, PhD, or equivalent experience.
- 15+ years of experience building large-scale distributed systems, high-performance storage, or ML systems infrastructure in C/C++ and Python, with a record of delivering production services.
- Deep understanding of memory hierarchies including GPU HBM, host DRAM, SSD, and remote or object storage, and experience designing systems spanning multiple tiers for performance and cost efficiency.
- Experience with distributed caching or key-value systems, particularly designs optimized for low latency and high concurrency.
- Hands-on experience with networked I/O and RDMA/NVMe-oF/NVLink-style technologies, and familiarity with disaggregated and aggregated deployments for AI clusters.
- Strong skills profiling and optimizing systems across CPU, GPU, memory, and network, using metrics to guide architectural decisions and validate improvements in TTFT and throughput.
- Excellent communication skills and prior experience leading cross-functional efforts with research, product, and customer teams.
Nice To Haves
- Contributions to open-source LLM serving or systems projects focused on KV-cache optimization, compression, streaming, or reuse.
- Experience designing unified memory or storage layers that provide a single logical KV or object model across GPU, host, SSD, and cloud tiers, particularly in enterprise or hyperscale environments.
- Publications or patents in LLM systems, memory-disaggregated architectures, RDMA/NVLink-based data planes, or KV-cache/CDN-like systems for ML.
Responsibilities
- Design and evolve a unified memory layer spanning GPU memory, pinned host memory, RDMA-accessible memory, SSD tiers, and remote file, object, and cloud storage for large-scale LLM inference.
- Architect and implement integrations with LLM serving engines such as vLLM, SGLang, and TensorRT-LLM, emphasizing KV-cache offload, reuse, and remote sharing across heterogeneous and disaggregated clusters.
- Co-design interfaces and protocols for disaggregated prefill, peer-to-peer KV-cache sharing, and multi-tier KV-cache storage.
- Partner with GPU architecture, networking, and platform teams to use GPUDirect, RDMA, NVLink, and similar technologies for low-latency KV-cache access and sharing.
- Mentor engineers and set technical direction for memory and storage subsystems.
- Represent the team in internal reviews and external forums, including open-source communities, conferences, and customer-facing technical deep dives.
Benefits
- Comprehensive benefits package.
- Eligibility for equity.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
