Senior Product Manager – AI Inference Performance

NVIDIA•Santa Clara, CA

10d•$208,000 - $327,750

About The Position

NVIDIA is seeking a highly technical Product Manager to own products that help customers maximize the performance of AI models and applications running on NVIDIA hardware. The role covers the inference stack, including optimization techniques, frameworks, benchmarking, and operational tooling. The Product Manager will turn successful internal deployment practices into broadly adoptable capabilities and drive NVIDIA’s inference performance strategy across deployments ranging from single-GPU workstations to large data centers. The role sets the inference performance roadmap, builds extensible platforms for different model families and deployment sizes, and defines strategies for agentic and multi-turn workloads. It also shapes how optimizations are delivered through TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo, working with open-source communities and internal engineering teams. The Product Manager owns benchmark methodology and performance claims, as well as product release readiness, quality, regression tracking, customer-blocking issues, and feedback from production deployments. The role requires independent leadership, technical depth, and the ability to communicate performance improvements in terms of business value.

Requirements

12 or more years in product management at a technology company, or comparable experience as a founder, engineering lead, or technical product owner.
Hands-on AI/LLM inference performance optimization experience in an engineering or product management role.
Deep familiarity with techniques such as KV caching and reuse, quantization, speculative decoding, and disaggregated serving.
Familiarity with TensorRT-LLM, vLLM, SGLang, NVIDIA Dynamo, and the surrounding inference serving and scheduling ecosystem.
Ability to work independently, define strategy for ambiguous problem spaces, and drive work to shipped outcomes.
Operational experience running a live product, including release management, quality and regression rigor, customer issues, and support processes.
Ability to explain low-level capabilities in terms of business value, including lower total cost of ownership, faster response, and improved GPU utilization.
BS, MS, or PhD in Computer Science, Computer Engineering, or another relevant area of study, or equivalent experience.

Nice To Haves

Engineering experience with LLM inference performance, including profiling, kernel-level or serving-level optimization, or building a serving stack.
Open-source contributions or product leadership in vLLM, SGLang, TensorRT-LLM, Triton Inference Server, or Dynamo.
Production experience at scale, including capacity planning, autoscaling, SLA management, or stateful multi-turn applications.
Habit of reading relevant research and translating it into roadmap decisions, with intuition for future model architectures and serving techniques.

Responsibilities

Own the inference performance roadmap, setting direction for model representation, memory and state management, request scheduling and serving, and token generation.
Evaluate emerging inference techniques and decide which to build, adopt, or retire.
Build platform capabilities that generalize across model families, deployment topologies, and customer sizes, with easy adoption, sensible defaults, and extensibility.
Define performance strategy for agentic applications and multi-turn workloads, including cross-turn cache reuse, request prioritization, and efficient handling of idle time in agent loops.
Define how optimizations are delivered across TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo, collaborating with open-source communities and internal engineering teams.
Own how performance is measured, published, and reproduced; define benchmark methodology, metrics, and credibility guardrails.
Manage day-to-day product operations, including release readiness, quality standards, regression tracking, customer-blocking issues, and production feedback into the roadmap.
Translate low-level product capabilities into business value for technical and executive audiences.