Senior Machine Learning Engineer, Quantized Inference

NVIDIA•Redmond, WA

About The Position

We are now looking for a Senior Machine Learning Engineer for Quantized Inference! NVIDIA is seeking machine learning engineers to accelerate the discovery and deployment of efficient inference recipes for LLMs. A recipe defines which operators are transformed into low-precision or sparsified variants unlocking throughput and latency gains without regressing accuracy nor verbosity. Recipes may incorporate techniques such as rotations, block scaling to attenuate outlier impact, or improved calibration data drawn from SFT/RL pipelines. Pushing the frontier of inference efficiency requires a holistic view of the workload. The candidate will navigate the full design space: identifying which layers are sensitive to quantization relative to their inference cost, diagnosing why specific recipes fail, and adapting training techniques such as quantization-aware distillation or targeted fine-tuning to recover accuracy where needed. Our team develops quantized and sparse recipes that ship and run at scale across NVIDIA's LLM product portfolio. Our recipes directly determine the cost and latency of serving models to millions of users. We collaborate with inference framework teams (vLLM, TRT-LLM) to ensure recipes translate into real throughput gains, and with post-training teams to source calibration data and co-design quantization-aware training curricula.

Requirements

Proficient in Python and PyTorch
Experience with quantization, sparsity, or other model compression techniques
Ability to design and run rigorous experiments: controlled ablations, statistical significance, reproducibility
Familiarity with LLM evaluation methodology (benchmarks, human-preference proxies, verbosity metrics)
MS/PhD in Computer Science, Computer Engineering, Machine Learning, or equivalent experience.
3+ years of experience in an applied ML role
Demonstrated ability to move fast with ambiguous requirements, with strong written and verbal communication

Nice To Haves

Published work or production experience in post-training quantization or quantization-aware training
Experience with SFT, RLHF/DPO, or distillation pipelines
Familiarity with inference serving frameworks (vLLM, TRT-LLM, SGLang)
Track record of debugging numerical issues in mixed-precision training or inference

Responsibilities

Prototype state-of-the-art quantization and sparsity recipes applied to LLM workloads
Design and execute post-training quantization or quantization-aware distillation experiments: prepare SFT/RL calibration datasets, manage checkpoint-level eval sweeps, and iterate on recipes based on results
Run accuracy and verbosity evaluations of quantized/sparsified LLM workloads at cluster scale
Develop data analysis tooling and visualizations for numerics debugging
Participate in code reviews and incorporate feedback
Contribute improvements upstream to open-source inference and optimization libraries; publish findings at ML conferences where appropriate