Principal Software Engineer - AI Runtime & Distributed Inference Systems

International Recruiting LLC•San Jose, CA

About The Position

This Principal Software Engineer will define and lead the software architecture for a large-scale AI inference platform powered by custom AI accelerators. The role focuses on transforming hundreds to thousands of accelerator ASICs into a unified system that serves dense and Mixture-of-Experts (MoE) foundation models at datacenter scale. The engineer will shape distributed runtimes, communication libraries, scheduling systems, memory management, and model-serving infrastructure to improve throughput, latency, efficiency, reliability, and programmability. The role spans distributed systems, AI infrastructure, networking, compilers, and computer architecture, and includes cross-functional leadership with silicon architecture, networking, compiler, runtime, infrastructure, and AI research teams. The position also drives production readiness, operational excellence, performance engineering, fleet management, and hardware-software co-design. Success over the first 12–24 months includes delivering strong TTFT and TPOT performance, improving accelerator utilization and throughput per rack and per watt, enabling scalable dense and MoE model execution, reducing communication overhead, establishing a robust production-ready runtime platform, and informing future hardware generations.

Requirements

BS, MS, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or a related field.
12+ years of software engineering experience.
5+ years building distributed systems at scale.
Strong expertise in C++, Rust, or similar systems programming languages.
Deep understanding of large-scale AI inference systems.
Experience with networking and distributed communication systems.
Proven track record leading architecture across multiple engineering teams.
Experience driving projects from concept through production deployment.

Nice To Haves

Experience with large language model serving systems.
Experience with vLLM, SGLang, TensorRT-LLM, Triton, PyTorch Distributed, NCCL, or equivalent technologies.
Experience designing runtime systems for custom accelerators, GPUs, TPUs, or AI ASICs.
Knowledge of transformer architectures, sparse models, and MoE systems.
Experience with RDMA, InfiniBand, RoCE, PCIe, Ethernet fabrics, or custom interconnects.
Experience in compiler technologies, graph optimization, or kernel scheduling.
Familiarity with HBM, memory hierarchy design, and cache management strategies.
Experience with heterogeneous accelerator programming models such as CUDA, HIP, or Triton, including device and memory management, kernel launching, error handling and diagnostics, and kernel optimization techniques such as tiling and fused kernels.
Experience implementing virtualization software, rack controllers, PCIe or memory controllers, and device lifecycle management software.
Experience with NCCL, RCCL, or MPI collective communications.
Experience with scale-up networking such as UALoE, ESUN, or NVLink; scale-out networking such as RoCEv2; and network congestion-control techniques such as PFC or DCQCN.

Responsibilities

Define the software architecture for large-scale AI inference systems.
Lead the evolution of the distributed runtime stack for dense and MoE foundation models.
Establish long-term technical strategy for performance, scalability, reliability, and programmability.
Influence hardware-software co-design decisions across future accelerator generations.
Architect runtime systems supporting tensor, pipeline, expert, and data parallelism, as well as multi-rack inference execution.
Design systems for efficient prefill and decode execution.
Drive continuous batching and request scheduling strategies.
Optimize time to first token (TTFT), time per output token (TPOT), throughput, utilization, and quality of service (QoS).
Lead design of token routing and expert execution systems.
Define expert placement and replication strategies.
Develop mechanisms for expert load balancing and congestion control.
Optimize communication and compute overlap across distributed expert clusters.
Architect high-performance collective communication libraries and design software interfaces for custom accelerator fabrics.
Develop scalable implementations of AllReduce, AllGather, ReduceScatter, Broadcast, and expert dispatch and gather.
Drive latency and bandwidth optimization across accelerator clusters.
Define architecture for distributed KV-cache management and drive memory placement, eviction, and tiering policies.
Optimize utilization across local memory, pooled memory, and storage tiers.
Create runtime services for dynamic memory allocation and workload balancing.
Establish performance methodology, benchmarks, and KPIs; analyze bottlenecks across compute, memory bandwidth, communication fabric, expert routing, and scheduling layers.
Lead root-cause analysis of production performance issues.
Drive reliability, observability, telemetry, and debugging architecture.
Define health monitoring and fleet management capabilities.
Partner with infrastructure teams to operate inference systems at scale.
Establish production readiness standards and operational excellence practices.
Deliver industry-leading TTFT and TPOT performance, high accelerator utilization, scalable execution of frontier dense and MoE models, and improved throughput per rack and per watt.
Reduce communication overhead, establish a robust production-ready AI runtime platform, and provide software-driven architectural insights for future hardware generations.