Principal Software Engineer, Inference

Hewlett Packard Enterprise•Durham, TX

2d•Hybrid

About The Position

HPE's Private Cloud AI organization is seeking a Principal Software Engineer to lead the model runtime within HPE AI Essentials, the inference platform used by enterprises to operate large language models on infrastructure they own, including air-gapped and sovereign environments. The principal engineering challenge in this domain is not model deployment but sustained execution efficiency: achieving low tail latency and high GPU utilization on customer-owned hardware of varying generation and configuration. In this role you will define the architecture of that runtime – engine integration, batching, KV cache management, and distributed execution – together with the Kubernetes orchestration layer that supports it. The primary work location is as listed, but could be any other HPE site location in the US; however, remote work options will be considered.

Requirements

Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals
Comprehensive understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding
Tensor and pipeline parallelism, NCCL collective operations, and the GPU memory hierarchy and interconnect characteristics that govern them
Expert level proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling
Strong programming proficiency in Go and Python, with the ability to read, debug, and profile C++/CUDA using tools such as Nsight
Experience with debugging/profiling multi-tier application workloads such as RAG, Agents, etc
Excellent analytical, debugging, and problem-solving abilities
Minimum of 12 years of experience in Software Engineering, including +1 years working directly on LLM inference runtimes or production model serving
Degree in Computer Science or related field

Nice To Haves

Upstream contribution to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache, or KServe
Disaggregated prefill/decode serving, or KV cache offload and reuse at scale
RDMA, GPUDirect Storage, InfiniBand, or RoCE
MIG, fractional GPU allocation, and multi-tenant GPU isolation
On-premises, air-gapped, or regulated enterprise software delivery

Responsibilities

Define and own the technical direction of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution
Partner with inference performance engineering teams, with accountability for time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency
Define distributed inferencing strategy, including disaggregated prefill/decode, tensor and pipeline parallelism, KV cache offload across GPU memory, host memory, and RDMA-attached storage
Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and determine whether each runtime is adopted, developed in-house, or declined
Define the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling
Mentor engineers, lead design and architecture reviews, and present technical direction to business unit and executive audiences