Senior Staff Software Engineer - AI/ML Inference Engineering

ServiceNow•San Diego, CA

23h•$181,200 - $317,100

About The Position

The AI Services Platform Group at ServiceNow builds intelligent software and smart user experiences using advanced technologies. The group includes researchers, applied scientists, engineers, and product managers, and works both to evolve the AI platform and to develop AI-powered products and work experiences. The Senior Staff Software Engineer will lead the architecture, optimization, and operational excellence of the AI inference platform. The role owns the end-to-end inference lifecycle, from onboarding models to deploying optimized inference engines in production across multiple backends on Kubernetes infrastructure. It also involves architecting an intelligent gateway layer using Envoy and ext_proc to enforce access control, provide observability, and support advanced request routing. The role combines expertise in GPU compute, LLM APIs, inference frameworks, containerization, orchestration, and API gateway architecture with a focus on pragmatic engineering and measurable performance impact.

Requirements

Expertise in Go or Python, object-oriented programming, design patterns, and time- and space-efficient algorithms.
Model-serving and runtime experience with systems such as vLLM, TensorRT-LLM, or Triton Inference Server.
Exposure to LLM APIs or ML-serving systems, or demonstrated ability to ramp up quickly.
Experience building new products that use challenging algorithms.
Expertise in writing efficient, object-oriented, modularized, high-quality software.
Knowledge of core AI/ML techniques and algorithms.
Knowledge of unit testing, profiling, and code tuning.
At least 8 years of backend software engineering experience.
At least 4 years of experience designing and operating distributed systems.
Bachelor’s degree in computer science, software engineering, or a closely related technical field; equivalent practical experience may be considered where technical depth is demonstrable.
Proficient use of AI coding assistants such as GitHub Copilot or Cursor, with careful review of generated code for logic errors, security anti-patterns, and edge cases.
Experience with AI-assisted debugging and trace analysis, and understanding of data privacy rules for information entered into prompts.

Nice To Haves

Experience deploying, scaling, and operating production Kubernetes services, including Helm-based deployments and CI/CD pipelines that support repeatable releases and safe rollbacks.
Experience owning observability, including metrics, logs, tracing, and customer-impact-focused alerts.
Experience independently resolving production incidents, participating in on-call, conducting root-cause analysis, and operating against service level objectives.
A master’s degree may offset up to one year of the experience minimum.

Responsibilities

Lead the architecture, optimization, and operational excellence of the AI inference platform.
Own the end-to-end inference lifecycle, from onboarding models to deploying optimized inference engines in production across multiple backends on Kubernetes.
Architect an intelligent gateway layer using Envoy and ext_proc to enforce access control, provide observability, and enable advanced request routing.
Deploy, scale, and operate services in production Kubernetes environments, including Helm deployments and CI/CD pipelines.
Build meaningful metrics, useful logs, tracing, and alerts focused on customer impact.
Debug and resolve production incidents, participate in on-call, conduct root-cause analysis, and operate against defined service level objectives.