Senior Staff Software Engineer - AI/ML Inference Engineering

ServiceNow•Kirkland, WA

22h•$201,300 - $352,300

About The Position

The Senior Staff Software Engineer - AI/ML Inference Engineering will lead the architecture, optimization, and operational excellence of ServiceNow’s AI inference platform. The role owns the end-to-end inference lifecycle, from onboarding new models to deploying optimized inference engines in production across multiple backends on Kubernetes infrastructure. It also architects an intelligent gateway layer using Envoy and ext_proc to enforce access control, provide observability, and support advanced request routing. The position combines expertise in GPU compute, LLM APIs, inference frameworks, containerization, orchestration, and API gateway architecture with pragmatic engineering and a focus on measurable performance impact. The AI Services Platform Group builds and evolves the AI platform, partners with teams on AI-powered products and work experiences, and researches and evaluates technologies for future experiences.

Requirements

Expertise in Go or Python, object-oriented programming, design patterns, and time- and space-efficient algorithms.
Model-serving and runtime experience, such as vLLM, TensorRT-LLM, or Triton Inference Server.
Exposure to LLM APIs or ML-serving systems, or a clear ability to ramp up quickly.
Experience building new products that use challenging algorithms.
Expertise in writing efficient, object-oriented, modularized, high-quality software.
Knowledge of core AI/ML techniques and algorithms.
Knowledge of unit testing, profiling, and code tuning.
Confident use of AI coding assistants such as GitHub Copilot or Cursor, with rigorous review of generated code for logic errors, security anti-patterns, and edge cases.
Experience with AI-assisted debugging and trace analysis, and understanding of data privacy rules for information included in prompts.
At least 8 years of backend software engineering experience.
At least 4 years of experience designing and operating distributed systems.
Bachelor’s degree in computer science, software engineering, or a closely related technical field; equivalent practical experience may be considered where technical depth is demonstrable. A master’s degree may offset up to one year of the experience minimum.

Nice To Haves

Experience deploying, scaling, and operating services in production Kubernetes environments, including Helm-based deployments and CI/CD pipelines that support repeatable releases and safe rollbacks.
Experience owning service observability, including metrics, logs, tracing, and alerts focused on customer impact.
Experience independently resolving production incidents, participating in on-call, performing root-cause analysis, and working against service level objectives.

Responsibilities

Lead the architecture, optimization, and operational excellence of the AI inference platform.
Own the inference lifecycle, from onboarding models to deploying optimized inference engines in production across multiple backends on Kubernetes.
Architect the intelligent gateway layer using Envoy and ext_proc for access control, observability, and advanced request routing.
Deploy, scale, and operate services in production Kubernetes environments, including Helm-based deployments and CI/CD pipelines.
Build observability for owned systems using meaningful metrics, useful logs, tracing, and customer-impact-focused alerts.
Independently debug and resolve production incidents, participate in on-call, conduct root-cause analysis, and operate against service level objectives.