Senior Staff Software Engineer - AI/ML Inference Engineering

ServiceNow•Santa Clara, CA

3d•$181,200 - $317,100

About The Position

The AI Services Platform Group at ServiceNow is a customer-focused innovation group of researchers, applied scientists, engineers, and product managers. The team builds and evolves the AI platform, partners with other teams on AI-powered products and work experiences, and researches and tests technologies that may enable future experiences. The Senior Staff Software Engineer will lead the architecture, optimization, and operational excellence of ServiceNow’s AI inference platform. The role owns the end-to-end inference lifecycle, from onboarding models to deploying optimized inference engines in production across multiple backends on Kubernetes. It also architects an intelligent gateway layer using Envoy and ext_proc for access control, observability, and advanced request routing. The position requires expertise across GPU compute, LLM APIs, inference frameworks, containerization, orchestration, and API gateway architecture, with an emphasis on pragmatic engineering and measurable performance impact.

Requirements

At least 8 years of backend software engineering experience.
At least 4 years designing and operating distributed systems.
Bachelor’s degree in computer science, software engineering, or a closely related technical field; equivalent practical experience may be considered where technical depth is clearly demonstrable.
A master’s degree may offset up to one year of the experience minimum.
Expertise in Go or Python, object-oriented programming, design patterns, and time- and space-efficient algorithms.
Model-serving and runtime experience, such as vLLM, TensorRT-LLM, or Triton Inference Server.
Exposure to LLM APIs or ML-serving systems, or a clear ability to ramp up quickly.
Experience building new products that use challenging algorithms.
Expertise in writing efficient, object-oriented, modularized, high-quality software.
Knowledge of core AI/ML techniques and algorithms.
Knowledge of unit testing, profiling, and code tuning.
Proficiency using AI coding assistants such as GitHub Copilot or Cursor, with the ability to rigorously review generated code for logic errors, security anti-patterns, and missed edge cases.
Ability to use AI-assisted debugging and trace analysis, and understanding of data privacy rules governing information included in prompts.

Nice To Haves

Experience deploying, scaling, and operating services in production Kubernetes environments, including Helm-based deployments and CI/CD pipelines.
Experience owning service observability through meaningful metrics, useful logs, tracing, and customer-impact-focused alerts.
Experience independently resolving production incidents, participating in on-call, performing root-cause analysis, and operating against service level objectives.

Responsibilities

Lead the architecture, optimization, and operational excellence of the AI inference platform.
Own the end-to-end inference lifecycle, including onboarding models and deploying optimized inference engines in production across multiple backends on Kubernetes.
Architect an intelligent gateway layer using Envoy and ext_proc to enforce access control, provide observability, and enable advanced request routing.
Deploy, scale, and operate services in production Kubernetes environments, including Helm deployments and CI/CD pipelines that support repeatable releases and safe rollbacks.
Own observability for services, including metrics, logs, tracing, and alerts focused on customer impact.
Independently debug and resolve production incidents, participate in on-call, conduct root-cause analysis, and operate against defined service level objectives.