Machine Learning Engineer, Inference & Serving (Speech LLM)

IntelliPro Group Inc. • San Francisco, CA

4d • $170,000 - $320,000 • Hybrid

About The Position

Build and deploy high-throughput, ultra-low-latency inference systems for large language models and foundational speech models. The role focuses on optimizing latency, throughput, and Time-To-First-Token or Time-To-First-Audio in real-time streaming environments, and on developing systems that understand and generate speech. It sits at the intersection of the core ML training and backend infrastructure teams and involves systems engineering to maximize performance across GPU clusters.

The position is based in San Francisco, California, with a hybrid schedule requiring at least three in-office days per week. Compensation is $170,000–$320,000 base salary, plus a performance bonus and equity.

Requirements

• Hands-on experience building and deploying high-throughput, ultra-low-latency inference engines for large language models or foundational speech models.
• Understanding of latency, throughput, and Time-To-First-Token or Time-To-First-Audio tradeoffs in real-time streaming environments.
• Practical experience with continuous batching, KV cache management (such as PagedAttention), and stateful connections for real-time conversational AI.
• Deep understanding of GPU architectures, including NVIDIA Ampere or Hopper, and memory hierarchy.
• Clear communication and effective collaboration skills.
• Enjoys systems engineering and optimizing performance across GPU clusters.

Nice To Haves

• Under-the-hood familiarity with LLM serving frameworks such as vLLM, TensorRT-LLM, SGLang, or NVIDIA Triton Inference Server; open-source contributions to these projects are a bonus.
• Experience handling continuous audio streams over WebSockets or WebRTC, deploying neural audio codecs, and managing chunked audio generation to minimize conversational latency.
• Experience implementing speculative decoding, lookahead decoding, or chunked prefill.
• Hands-on experience with post-training quantization and deploying models in FP8, INT8, AWQ, or GPTQ without degrading audio naturalness or ASR accuracy.
• Experience deploying multi-GPU tensor-parallel and multi-node inference pipelines, and managing autoscaling infrastructure using Kubernetes.

Responsibilities

• Build and deploy high-throughput, ultra-low-latency inference systems for large language models and foundational speech models.
• Optimize latency, throughput, and Time-To-First-Token or Time-To-First-Audio for real-time streaming.
• Apply continuous batching, KV cache management, and stateful connections for real-time conversational AI.
• Identify and eliminate hardware bottlenecks across GPU architectures and memory hierarchies.
• Collaborate across the core ML training and backend infrastructure teams.
• Develop AI systems that understand and generate speech.

Benefits

• Performance bonus and equity.
• Healthcare for employees and dependents, including dental and vision coverage, with an employer subsidy.
• 401(k) plan with company matching for full-time employees.
• Unlimited paid time off.
• 13 paid holidays.
• 12 weeks of paid new-parent leave, regardless of gender.
• Choice of top-of-the-line laptops or workstations.
• Annual offsites.
• Fully stocked office.

What This Job Offers

• Job Type: Full-time
• Education Level: No Education Listed