Senior Staff Software Engineer - AI/ML Inference Engineering

ServiceNow•Santa Clara, CA

About The Position

The AI Services Platform Group at ServiceNow builds intelligent software and smart user experiences using advanced technologies. Its researchers, applied scientists, engineers, and product managers develop and evolve the AI platform, partner with teams on AI-powered products and experiences, and research and test technologies that may enable future work experiences. The Senior Staff Software Engineer will lead the architecture, optimization, and operational excellence of ServiceNow’s AI inference platform. The role owns the end-to-end inference lifecycle, from onboarding new models to deploying optimized inference engines in production across multiple backends on Kubernetes. It also architects an intelligent gateway layer using Envoy and ext_proc to enforce access control, provide observability, and support advanced request routing. The work combines expertise in GPU compute, LLM APIs, inference frameworks, containerization, orchestration, and API gateway architecture with a focus on pragmatic engineering and measurable performance.

Requirements

Expertise in Go or Python, object-oriented programming, design patterns, and time- and space-efficient algorithms.
Model-serving and runtime experience with systems such as vLLM, TensorRT-LLM, or Triton Inference Server.
Exposure to LLM APIs or ML-serving systems, or the ability to ramp up quickly.
Experience building new products that use challenging algorithms.
Expertise in writing efficient, object-oriented, modularized, high-quality software.
Knowledge of core AI/ML techniques and algorithms.
Knowledge of unit testing, profiling, and code tuning.
Confident use of AI coding assistants such as GitHub Copilot or Cursor, with the ability to critically review AI-generated code for logic errors, security anti-patterns, and edge cases.
Experience with AI-assisted debugging and trace analysis, and understanding of data privacy rules for information used in prompts.
At least 8 years of backend software engineering experience.
At least 4 years designing and operating distributed systems.
Bachelor’s degree in computer science, software engineering, or a closely related technical field. Equivalent practical experience may be considered where technical depth is clearly demonstrable.
Ability to deploy, scale, and operate services in production Kubernetes environments, including Helm-based deployments and repeatable, safely reversible CI/CD releases.
Ability to debug and resolve production incidents independently, participate in on-call, perform root-cause analysis, and work against defined service level objectives.

Nice To Haves

Experience with model-serving and runtime systems such as vLLM, TensorRT-LLM, or Triton Inference Server.
Exposure to LLM APIs or ML-serving systems.
A master’s degree, which may offset up to one year of the experience minimum.

Responsibilities

Lead the architecture, optimization, and operational excellence of the AI inference platform.
Own the end-to-end inference lifecycle, including onboarding models and deploying optimized inference engines in production across multiple backends on Kubernetes.
Architect the intelligent gateway layer using Envoy and ext_proc to enforce access control, provide observability, and enable advanced request routing.
Deploy, scale, and operate services in production Kubernetes environments, including Helm-based deployments and CI/CD pipelines.
Build observability for services, including metrics, logs, tracing, and alerts focused on customer impact.
Independently debug and resolve production incidents, participate in on-call, conduct root-cause analysis, and operate against defined service level objectives.