Software Engineer

ServiceNow•Santa Clara, CA

2d•$125,700 - $194,800•Hybrid

About The Position

The Build Agent is ServiceNow's AI coding assistant, purpose-built for the platform's metadata-driven substrate - operating natively across ServiceNow's scoped applications, tables, and metadata types. The framework is critical to Build Agent success — it's already driven measurable wins: recovering Build Agent correctness, cutting latency and inference cost ahead of a major release, and catching high-severity defects that manual testing missed before they reached production. Key areas of focus include: Evaluation Infrastructure: Golden prompt sets, Pass@1 and functional scoring, failure categorization (plumbing vs. metadata-creation failures), and coverage across ServiceNow metadata types and UI workflows. Model Support & Benchmarking: Structured evaluation of candidate foundation models against the production default, with failure-consistency analysis to separate scaffold/tuning issues from genuine capability gaps. Telemetry: Token usage, cache efficiency, and inference cost tracked alongside correctness as first-class release signals. What you get to do in this role: Eval orchestration at scale, Agent observability and tracing.

Requirements

Direct experience building evaluation systems for LLM-based products — scoring methodology, benchmark design, and failure analysis, not just prompt engineering.
A working understanding of coding agent architecture: tool use, agentic loops, context management, and the tradeoffs between generic and domain-specific scaffolding.
2+ years of experience with technologies relevant to ServiceNow, including advanced coding skills and fluency in one or more of Java, C++, Shell, or JavaScript.
Experience critically evaluating foundation models — distinguishing real capability differences from tuning or scaffold artifacts.
Ability to execute against ambiguous priorities, weighing context, risk, and desired outcomes — particularly where eval data is incomplete or contested.