Principal Software Engineer, Inference
About The Position
Hewlett Packard Enterprise’s Private Cloud AI organization is seeking a Principal Software Engineer to lead the model runtime within HPE AI Essentials, an inference platform for enterprises operating large language models on customer-owned infrastructure, including air-gapped and sovereign environments. The role focuses on sustained execution efficiency across varied hardware, including low tail latency and high GPU utilization. The engineer will define the runtime architecture—including engine integration, batching, KV cache management, and distributed execution—and the supporting Kubernetes orchestration layer. The role is designed as hybrid, with an average of two days per week in an HPE office. The primary work location is the location listed in the posting, but another HPE site in the US may be considered; remote work options will also be considered. The position calls for at least 12 years of Software Engineering experience, including more than one year working directly on LLM inference runtimes or production model serving, and a degree in Computer Science or a related field. The role includes technical leadership, mentoring, design and architecture reviews, and communicating technical direction to business unit and executive audiences.
Requirements
- Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modifying engine internals.
- Comprehensive understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding.
- Knowledge of tensor and pipeline parallelism, NCCL collective operations, and GPU memory hierarchy and interconnect characteristics.
- Expert-level proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling.
- Strong programming proficiency in Go and Python, and ability to read, debug, and profile C++/CUDA using tools such as Nsight.
- Experience debugging and profiling multi-tier application workloads such as RAG and agents.
- Excellent analytical, debugging, and problem-solving abilities.
- Minimum of 12 years of Software Engineering experience, including more than one year working directly on LLM inference runtimes or production model serving.
- Degree in Computer Science or a related field.
Nice To Haves
- Upstream contributions to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache, or KServe.
- Experience with disaggregated prefill/decode serving, or KV cache offload and reuse at scale.
- Experience with RDMA, GPUDirect Storage, InfiniBand, or RoCE.
- Experience with MIG, fractional GPU allocation, and multi-tenant GPU isolation.
- Experience delivering on-premises, air-gapped, or regulated enterprise software.
Responsibilities
- Define and own the technical direction of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution.
- Partner with inference performance engineering teams and be accountable for time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency.
- Define distributed inferencing strategy, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage.
- Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and decide whether each runtime should be adopted, developed in-house, or declined.
- Define the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling.
- Mentor engineers, lead design and architecture reviews, and present technical direction to business unit and executive audiences.
Benefits
- Comprehensive benefits intended to support employees’ physical, financial, and emotional wellbeing.
- Programs supporting personal and professional development and career goals.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
