Staff Software Engineer, LLM Inference Platform

Zenlayer, Inc•Cupertino, CA

7h•$300,000 - $600,000

About The Position

As the technical lead for inference performance, you will own end-to-end responsibility for the platform's Token Performance. For models ranging from hundreds of billions to trillions of parameters, including MoE, you will continuously reduce cost per million tokens and increase per-GPU Goodput while meeting TTFT/TPOT SLOs, and prove the gain from every optimization with reproducible data. You will establish performance metrics and production-representative benchmarks; attribute bottlenecks across inference engines and models; optimize inference engines and kernels; lead Prefill-Decode disaggregation and KV cache transfer design; develop MoE parallelism strategies; and evaluate quantization trade-offs across engines and hardware.

Requirements

First-principles understanding of Transformer inference, including the compute-versus-memory-bound nature of Prefill and Decode, the KV Cache memory model, differences among MHA/GQA/MLA, and how MoE routing constrains parallelism.
Source-level, in-depth experience with at least one mainstream inference engine, including real cases of identifying and fixing bottlenecks in scheduling, kernel selection, or KV management.
CUDA kernel tuning skills, including Nsight profiling, memory optimization, fusion, and CUDA Graph, with hands-on practical experience.
Bachelor’s degree in Computer Science, Electrical/Computer Engineering, or a related field such as HPC systems or ML Systems.
Minimum 6 years in systems/Performance Engineering, with at least 2–3 years directly on LLM or GPU performance work.

Nice To Haves

Experience with merged PRs in vLLM.
Experience operating at cluster scale (hundreds to thousands of GPUs).

Responsibilities

Establish a unified Token Performance metrics framework covering TTFT, TPOT, Goodput, GPU utilization, and per-token cost, and build load-testing benchmarks reflecting production traffic, including high concurrency, long context, and multi-turn Agent workloads.
Attribute bottlenecks for models such as DeepSeek, Kimi, GLM, Qwen, and MiniMax across compute, memory bandwidth, communication, scheduling, and queuing.
Customize and contribute to vLLM, SGLang, or TensorRT-LLM on critical paths such as the scheduler, Continuous Batching, Chunked Prefill, CUDA Graph, and KV block management.
Define responsibility boundaries between the inference engine and orchestration layers such as Dynamo and llm-d, and reduce tail latency through request routing and load balancing.
Lead the core design of Prefill-Decode disaggregation, including the KV transfer path (NIXL/RDMA/UCX), transfer-compute overlap, and phase-aware autoscaling.
Define failure domains and recovery semantics for cross-instance KV transfer, and determine which workloads benefit from PD disaggregation and which may be slower.
Design hybrid TP/PP/DP/EP parallelism strategies and quantify trade-offs between EP and TP.
Address expert load imbalance and All-to-All communication overhead, and drive Wide-EP into production.
Write and tune critical CUDA and ROCm/HIP operators, including memory access optimization, kernel fusion, and Attention-, MoE-, and communication-related kernels.
Build a closed-loop quantization–accuracy–throughput evaluation of FP8/FP4/INT4/AWQ/GPTQ across different engines and hardware.
Reduce cost per million tokens and increase per-GPU Goodput while meeting TTFT/TPOT SLOs, demonstrating optimization gains with reproducible data.