LLM Inference Engineer

Health GPT Inc • Palo Alto, CA

44d • Onsite

About The Position

We're seeking an experienced LLM Inference Engineer to optimize our large language model (LLM) serving infrastructure. The ideal candidate has: Extensive hands-on experience with state-of-the-art inference optimization techniques A track record of deploying efficient, scalable LLM systems in production environments

Requirements

• Experience optimizing LLM inference systems at scale
• Proven expertise with distributed serving architectures for large language models
• Hands-on experience implementing quantization techniques for transformer models
• Strong understanding of modern inference optimization methods, including:
• Speculative decoding techniques with draft models
• Eagle speculative decoding approaches
• Proficiency in Python and C++
• Experience with CUDA programming and GPU optimization

Nice To Haves

• Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM
• Experience with custom CUDA kernels
• Track record of deploying inference systems in production environments
• Deep understanding of performance optimization systems

Responsibilities

• Design and implement multi-node serving architectures for distributed LLM inference
• Optimize multi-LoRA serving systems
• Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality
• Implement speculative decoding and other latency optimization strategies
• Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases
• Continuously benchmark and improve system performance across various deployment scenarios and GPU types

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed
• Number of Employees: 101-250 employees