Staff Machine Learning Engineer, Agent Eval Platform

ServiceNow•Mountain View, CA

16h

About The Position

Moveworks is hiring a Staff Machine Learning Engineer to build the evaluation and improvement loop for AI agents that plan, call tools, and take actions in enterprise systems. The role focuses on evaluating and improving the agent harness—not training the underlying model—including instructions, tool descriptions, routing, knowledge, and memory. It combines applied machine learning and systems engineering, with an emphasis on trustworthy measurements that can be reproduced across hundreds of customer organizations. The engineer will design calibrated evaluation methods, separating deterministic checks of world state from LLM-based judgments of subjective qualities. They will build confidence-aware scoring, maintain calibration against human-labeled trajectories, address shared blind spots between simulators and judges, and investigate differences between offline simulation and production. The role also includes creating a self-improving harness that identifies real-world failures, generates realistic evaluation scenarios and environments, determines where fixes belong, proposes and checks improvements on held-out cases, and turns resolved failures into permanent tests. Organization-specific learnings must remain appropriately scoped, with admin approval for org-specific changes. At the Staff level, the engineer will own the measurement architecture for this loop end to end, establish the organization's standard for deciding whether changes ship, partner technically with the agent-building team, and mentor engineers on evaluation methodology. The role is with Moveworks, which was acquired by ServiceNow in December 2025.

Requirements

At least 5 years of experience in applied ML or ML-adjacent engineering, with systems shipped and operated for others to rely on.
Experience turning subjective human judgment into a validated measurement that teams use to gate decisions.
Experience improving a multi-component system through measurement, such as pipelines, retrieval, prompts, policies, or routing, rather than only changing model weights.
Strong applied ML fundamentals and comfort evaluating and prompting LLMs rather than training them.
Strong Python and production engineering skills, including services, data pipelines, and reproducibility practices such as versioned data, deterministic replay, and recorded inputs.
Ability to reason and communicate clearly about complex problems and explain what measurements mean.
High ownership and a bias toward shipping at startup pace.
Comfort with ambiguity and sound judgment about when a measurement is reliable enough to act on.
Experience in at least three of the following: LLM-as-judge or automated evaluation design and human calibration; human annotation programs, rubric authoring, label quality, or annotator throughput; rigorous experimentation using held-out sets, pre-established decision rules, variance and noise analysis, or regression detection; large-scale failure analysis using logs or traces; replay, simulation, or sandboxed test infrastructure; synthetic evaluation data and seeded environments with quality controls; versioned, tested, and measured prompt or harness optimization; search ranking, recommendation systems, or online-experiment evaluation; high-volume trace or event-data pipelines.
At Staff level, ability to own the self-improving loop's measurement architecture, establish organizational measurement standards for shipping decisions, partner with the agent-building team, and mentor engineers.

Nice To Haves

Experience evaluating LLM agents through multi-step trajectories, tool calls, and actions in stateful environments.
Experience automatically generating complete agent environments, including tasks, stateful worlds, and verifiers, at scale.
Published or shipped evaluation methodology adopted by others.
Built evaluation tooling used by teams outside one's own team or an evaluation product used by customers.
Experience with agentic systems that take actions rather than only generate text.

Responsibilities

Design a shared, configurable base judge and per-item evaluation rubrics.
Separate deterministic validators for checkable world-state outcomes from LLM judgments for subjective aspects of agent behavior.
Develop scoring that reports confidence and routes uncertain judgments to human review.
Maintain a calibration loop using human-labeled trajectories in partnership with the annotation team.
Guard against correlated blind spots between user simulators and LLM judges.
Investigate offline-versus-online evaluation divergence and keep evaluation suites refreshed with production failures.
Surface failures in real agent use, including failures that appear successful.
Generate realistic synthetic scenarios and seeded enterprise environments with expected outcomes and quality controls.
Determine whether agent failures call for changes to instructions, tools, routing, knowledge, or memory.
Optimize harness components against simulation and verify proposed changes on cases not used for tuning.
Detect regressions that aggregate scores might conceal and keep change proposal separate from change evaluation.
Convert fixed failures into permanent tests and ensure organization-specific learnings are scoped appropriately, with admin approval for org-specific changes.
Build reproducible evaluation infrastructure that operates at scale across hundreds of customer organizations.
At Staff level, own the end-to-end measurement architecture, set the standard for ship decisions, serve as the technical counterpart to the agent-building team, and mentor engineers on evaluation methodology.