Senior Software Engineer, AI Inference

NVIDIA • Toronto, ON

29d

About The Position

Help us push the boundaries of AI inference at NVIDIA — where your systems expertise shapes both the technology and the teams building on top of it! We're looking for a Senior Software Engineer to work at the frontier of large-scale LLM serving, partnering directly with some of the world's most technically demanding customers to unlock the full performance potential of NVIDIA's inference stack. In this role, you'll combine deep systems knowledge with hands-on customer engagement — profiling real deployments, benchmarking across GPU clusters, and turning insights into improvements that ripple across the open-source ecosystem. Do you love digging into performance problems that don't have obvious answers, and want your work to have an impact far beyond a single codebase? We'd love to talk. Unlike traditional customer-facing engineering roles, we expect you to go far deeper — contributing to vLLM, NVIDIA Dynamo, and the tooling that makes every engineer on your team more effective.

Requirements

• Bachelor's, Master's, or PhD in Computer Science, Computer Engineering, or equivalent experience.
• 5+ years of industry experience building and operating complex, production-grade software systems, with strong instincts for how systems behave at scale.
• Hands-on experience deploying and operating LLM inference workloads — particularly with vLLM — including configuration, optimization, and debugging in real-world environments.
• Proficiency with container orchestration (Kubernetes) and HPC scheduling (Slurm) for running GPU-accelerated workloads.
• Solid understanding of LLM serving fundamentals: batching strategies (continuous batching, chunked prefill), KV cache management, and tensor/pipeline parallelism.
• Familiarity with GPU performance analysis: memory hierarchy, utilization, roofline modeling, and profiling with Nsight Systems or Nsight Compute.
• Strong written and verbal communication skills, with the ability to present technical findings clearly to both engineering teams and leadership — and to navigate ambiguous, open-ended customer problems.

Nice To Haves

• Experience with NVIDIA Dynamo or other disaggregated inference serving frameworks.
• Contributions to open-source inference or ML systems projects, particularly vLLM or SGLang — please include links to relevant pull requests or artifacts.
• Background with ML compilers or GPU kernel development (Triton, CUTLASS, TorchInductor).
• Experience building developer tools or internal platforms that meaningfully improved team productivity.
• Prior experience in a customer-facing or forward-deployed engineering capacity within a technical product organization.

Responsibilities

• Work directly with customer engineering teams through long-term technical partnerships, understanding their LLM serving architectures and performance goals, then designing and implementing end-to-end benchmarking campaigns across Kubernetes and Slurm environments to surface actionable insights.
• Set up and operate vLLM serving deployments on GPU clusters, tuning configurations for throughput, latency, and efficiency — and collect Nsight Systems / Nsight Compute profiling traces to identify performance gaps relative to reference frameworks.
• Develop detailed performance plans based on profiling findings and collaborate with NVIDIA's kernel engineering and OSS vLLM teams to drive improvements that benefit both your customers and the broader community.
• Build internal tools, benchmarking harnesses, and automation pipelines that raise the productivity of your teammates and customers alike — with a multiplier attitude that makes everyone around you more effective.
• Document architectures, findings, and recommendations with clarity for technical audiences, and contribute improvements back to vLLM and related open-source projects where appropriate.

Benefits

• NVIDIA offers highly competitive salaries and a comprehensive benefits package.
• As you plan your future, see what we can offer to you and your family www.nvidiabenefits.com/

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Ph.D. or professional degree
• Number of Employees: 5,001-10,000 employees