Senior Staff Software Engineer

ServiceNow•San Diego, CA

20h•$171,900 - $300,800

About The Position

The Senior Staff Software Engineer will lead the team responsible for the evaluation framework, model support, and telemetry for ServiceNow’s Build Agent, an AI coding assistant designed to work natively across ServiceNow applications, tables, and metadata types. The team’s evaluation framework has helped improve correctness, reduce latency and inference cost, and identify high-severity defects before production. The role will help scale this work into a durable function across an eight-engineer team and multiple contributing teams. The position owns evaluation strategy and roadmap, including prompt set design, scoring methodology, failure-mode taxonomy, and telemetry instrumentation. It also leads foundation-model benchmarking and makes data-backed recommendations on model support. The role coordinates quality standards across teams, converts evaluation findings into owned fixes, and addresses ambiguous problems involving evaluation signals, model capabilities, and platform architecture.

Requirements

Direct experience building or leading evaluation systems for LLM-based products, including scoring methodology, benchmark design, and failure analysis.
Understanding of coding agent architecture, including tool use, agentic loops, context management, and trade-offs between generic and domain-specific scaffolding.
At least 6 years of experience with technologies relevant to ServiceNow.
Advanced coding skills and fluency in one or more of Java, C++, Ruby, Shell, or JavaScript.
Experience critically evaluating foundation models and distinguishing capability differences from tuning or scaffold artifacts.
Ability to execute against ambiguous priorities, weighing context, risk, and desired outcomes, especially when evaluation data is incomplete or contested.

Responsibilities

Own the evaluation strategy and roadmap, including prompt set design, scoring methodology, failure-mode taxonomy, and telemetry instrumentation.
Coordinate across teams contributing to Build Agent to set quality standards and drive evaluation findings to committed, owned fixes.
Lead model benchmarking and make data-backed recommendations on model support decisions.
Resolve ambiguous, cross-cutting problems involving evaluation signals, model capabilities, and platform architecture.
Develop evaluation infrastructure covering golden prompt sets, Pass@1 and functional scoring, failure categorization, and coverage across ServiceNow metadata types and UI workflows.
Support structured evaluation of candidate foundation models against the production default, including failure-consistency analysis.
Track token usage, cache efficiency, and inference cost alongside correctness as release signals.
Arbitrate evaluation coverage and define quality standards across surfaces owned by different contributing teams.