Principal Software Engineer, Inference
About The Position
Hewlett Packard Enterprise’s Private Cloud AI organization is seeking a Principal Software Engineer to lead the model runtime within HPE AI Essentials, an inference platform for enterprises operating large language models on infrastructure they own, including air-gapped and sovereign environments. The central challenge is sustaining execution efficiency on customer-owned hardware of varying generations and configurations, with low tail latency and high GPU utilization. The role defines the architecture of the inference runtime—including engine integration, batching, KV cache management, and distributed execution—and the Kubernetes orchestration layer that supports it. The role partners with inference performance engineering teams, guides technical decisions about emerging inference approaches, and provides technical leadership through mentoring, design reviews, and presentations. The position is hybrid, with an average requirement of two days per week in an HPE office. The primary work location is the location listed in the posting, though another HPE site in the United States may be considered; remote work options will also be considered.
Requirements
- Minimum of 12 years of experience in Software Engineering, including more than 1 year working directly on LLM inference runtimes or production model serving.
- Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals.
- Comprehensive understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding.
- Knowledge of tensor and pipeline parallelism, NCCL collective operations, and GPU memory hierarchy and interconnect characteristics.
- Expert-level proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling.
- Strong programming proficiency in Go and Python, with the ability to read, debug, and profile C++/CUDA using tools such as Nsight.
- Experience debugging and profiling multi-tier application workloads such as RAG and agents.
- Excellent analytical, debugging, and problem-solving abilities.
- Degree in Computer Science or a related field.
Nice To Haves
- Upstream contribution to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache, or KServe.
- Experience with disaggregated prefill/decode serving, or KV cache offload and reuse at scale.
- Experience with RDMA, GPUDirect Storage, InfiniBand, or RoCE.
- Experience with MIG, fractional GPU allocation, and multi-tenant GPU isolation.
- Experience delivering on-premises, air-gapped, or regulated enterprise software.
Responsibilities
- Define and own the technical direction of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution.
- Partner with inference performance engineering teams and be accountable for time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency.
- Define distributed inferencing strategy, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage.
- Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and determine whether each runtime is adopted, developed in-house, or declined.
- Define the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling.
- Mentor engineers, lead design and architecture reviews, and present technical direction to business unit and executive audiences.
Benefits
- A comprehensive suite of benefits supporting physical, financial, and emotional wellbeing.
- Personal and professional development programs to support career goals.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
