Senior Staff Software Engineer - AI/ML Inference Engineering

ServiceNow•Santa Clara, CA

21h•$201,300 - $352,300

About The Position

The Senior Staff Software Engineer will lead the architecture, optimization, and operational excellence of ServiceNow’s AI inference platform. The role owns the end-to-end inference lifecycle, from onboarding advanced models to deploying optimized inference engines in production across multiple backends on Kubernetes infrastructure. It also architects an intelligent gateway layer using Envoy and ext_proc to enforce access control, provide observability, and support advanced request routing. The position combines expertise in GPU compute, LLM APIs, inference frameworks, containerization, orchestration, and API gateway architecture with a focus on pragmatic engineering and measurable performance impact. The role is part of the AI Services Platform Group, a customer-focused team of researchers, applied scientists, engineers, and product managers. The group builds and evolves the AI platform, partners with teams on AI-powered products and work experiences, and researches and de-risks technologies that can enable future work experiences.

Requirements

Expertise in Go or Python, object-oriented programming, design patterns, and time- and space-efficient algorithms.
Experience with model serving and runtimes such as vLLM, TensorRT-LLM, or Triton Inference Server.
Exposure to LLM APIs or ML-serving systems, or demonstrated ability to ramp up quickly.
Experience building new products that use challenging algorithms.
Expertise in writing efficient, object-oriented, modular, high-quality software.
Knowledge of core AI/ML techniques and algorithms.
Knowledge of unit testing, profiling, and code tuning.
At least 8 years of backend software engineering experience.
At least 4 years designing and operating distributed systems.
Bachelor’s degree in computer science, software engineering, or a closely related technical field; equivalent practical experience may be considered where technical depth is clearly demonstrated.
Confident use of AI coding assistants such as GitHub Copilot or Cursor, with the ability to rigorously review AI-generated code for logic errors, security anti-patterns, and missed edge cases.
Experience with AI-assisted debugging and trace analysis, and understanding of data privacy rules governing prompt content.

Nice To Haves

A master’s degree may offset up to one year of the experience minimum.
Experience deploying, scaling, and operating production services on Kubernetes, including Helm and CI/CD pipelines.
Experience owning service observability, independently resolving production incidents, participating in on-call, conducting root-cause analysis, and working to service level objectives.

Responsibilities

Lead the architecture, optimization, and operational excellence of the AI inference platform.
Own the end-to-end inference lifecycle, including onboarding models and deploying optimized inference engines across multiple production backends on Kubernetes.
Architect an intelligent gateway layer using Envoy and ext_proc for access control, observability, and advanced request routing.
Deploy, scale, and operate services in production Kubernetes environments, including Helm-based deployments and repeatable, safely reversible CI/CD releases.
Build and maintain meaningful metrics, useful logs, tracing, and alerts focused on customer impact.
Independently debug and resolve production incidents, participate in on-call, conduct root-cause analysis, and operate against defined service level objectives.
Build and evolve the AI platform and partner with teams to develop AI-powered products and work experiences.
Use AI coding assistants and AI-assisted debugging and trace analysis while reviewing generated code critically and following data privacy rules for prompts.