Principal Software Engineer, Inference
About The Position
Hewlett Packard Enterprise’s Private Cloud AI organization is seeking a Principal Software Engineer to lead the model runtime within HPE AI Essentials, an inference platform for enterprises running large language models on infrastructure they own, including air-gapped and sovereign environments. The role focuses on sustained execution efficiency across customer-owned hardware of varying generations and configurations, including low tail latency and high GPU utilization. The engineer will define the runtime architecture—including engine integration, batching, KV cache management, and distributed execution—as well as the Kubernetes orchestration layer that supports it. The primary work location is as listed, with other HPE sites in the United States also possible. Remote work options will be considered, and the role is designated hybrid, requiring an average of two days per week at an HPE office. This position requires production experience with LLM inference engines and deep knowledge of inference internals, GPU execution, and Kubernetes platform architecture. The engineer will also mentor other engineers, lead architecture reviews, and communicate technical direction to business unit and executive audiences. HPE describes its benefits as supporting physical, financial, and emotional wellbeing and offers programs for personal and professional development.
Requirements
- At least 12 years of experience in software engineering, including more than 1 year working directly on LLM inference runtimes or production model serving.
- Degree in Computer Science or a related field.
- Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals.
- Comprehensive understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding.
- Knowledge of tensor and pipeline parallelism, NCCL collective operations, and GPU memory hierarchy and interconnect characteristics.
- Expert-level proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling.
- Strong programming proficiency in Go and Python, and ability to read, debug, and profile C++/CUDA using tools such as Nsight.
- Experience debugging and profiling multi-tier application workloads such as RAG and agents.
- Excellent analytical, debugging, and problem-solving abilities.
Nice To Haves
- Upstream contributions to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache, or KServe.
- Experience with disaggregated prefill/decode serving or KV cache offload and reuse at scale.
- Experience with RDMA, GPUDirect Storage, InfiniBand, or RoCE.
- Experience with MIG, fractional GPU allocation, and multi-tenant GPU isolation.
- Experience delivering on-premises, air-gapped, or regulated enterprise software.
Responsibilities
- Define and own the technical direction of LLM serving deployments, including engine integration, continuous batching, KV cache management and reuse, and quantized execution.
- Partner with inference performance engineering teams and be accountable for time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency.
- Define the distributed inference strategy, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage.
- Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and decide whether each is adopted, developed in-house, or declined.
- Define the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling.
- Mentor engineers, lead design and architecture reviews, and present technical direction to business unit and executive audiences.
Benefits
- Comprehensive benefits intended to support physical, financial, and emotional wellbeing for employees and their loved ones.
- Programs for personal and professional development and career goals.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
