Staff Machine Learning Engineer

ServiceNow•Santa Clara, CA

About The Position

The Staff AI Engineer will own significant parts of the agent harness, the infrastructure layer that enables AI agents to reason over enterprise data, act across workflows, and operate safely at Fortune 500 scale. The role is on ServiceNow’s Agent Orchestration team within the Agentic Engineering organization, which develops the execution core for conversational AI, including the agent harness, orchestration runtime, multi-agent coordination, memory management, and evaluation frameworks. The engineer will design and build orchestration infrastructure for multi-step agent workflows, ensure runtime reliability and visibility in production, create prompt management and evaluation systems, and integrate with frontier large language models. The role also includes technical leadership through architecture decisions, code reviews, coaching, and defining standards for agent and system-boundary design.

Requirements

At least 7 years of experience building production software systems, with a strong track record in reliability, performance, and scalability.
Hands-on experience shipping generative AI products and owning AI-powered features used by production users.
Solid understanding of how large language models work, including failure modes, context constraints, and how prompt design affects behavior at scale.
Experience with Python, Kubeflow, and SQL.
Practical prompt engineering experience, including systematically designing, versioning, and evaluating prompts across model updates or A/B evaluation cycles.
A track record in evaluation engineering, including designing and shipping evaluation suites used to make production AI quality decisions.
Experience reasoning about model routing, inference cost, and latency tradeoffs in production.
Strong software engineering fundamentals, including distributed systems, API design, and testing discipline.
Comfort working in fast-moving, ambiguous, startup-like AI product environments.

Nice To Haves

Experience with SageMaker, Vertex AI, or AzureML.
Experience with multi-agent coordination patterns such as A2A or MCP.
Familiarity with agent frameworks such as LangChain or LlamaIndex.
Prior experience shipping AI systems in enterprise software.
Experience with AI observability tooling, including tracing, cost tracking, and LLM-specific monitoring.
Familiarity with cloud-native infrastructure, service observability, logging, monitoring, reliability engineering, and production troubleshooting.

Responsibilities

Design and build the agent execution harness to route inputs, manage context, invoke tools, handle retries, and surface execution state across multi-step agentic workflows.
Own runtime fault tolerance, latency, and throughput for reliable enterprise workflows.
Instrument the harness with tracing, cost attribution, and latency visibility to monitor production agent behavior and identify failures.
Build prompt management systems for versioning, templating, and systematic evaluation across model updates and configuration changes.
Design and own unit evaluation, integration evaluation, and production monitoring frameworks to measure agent quality, catch regressions, and support data-informed decisions.
Integrate with and abstract over frontier LLMs, managing model routing, fallback strategies, cost, and latency tradeoffs in production.
Set technical standards through architecture decisions, code reviews, and coaching, particularly in agentic design patterns and production AI practices.
Define boundaries for agent logic, including decisions about tool calls, sub-agents, hardcoded paths, and human escalation, and establish design standards across the team.