Senior Machine Learning Engineer, Agent Eval Platform

ServiceNow•Santa Clara, CA

17h

About The Position

Moveworks is seeking a Senior Machine Learning Engineer to build and improve the evaluation platform and self-improving agent harness for AI agents that plan, call tools, and change state in enterprise systems. The role focuses on improving the system around the model—including instructions, tool descriptions, routing, knowledge, and memory—not training the underlying model. It blends applied machine learning and systems engineering, with an emphasis on producing trustworthy, reproducible signals across hundreds of customer organizations. The engineer will develop calibrated evaluation methods that combine deterministic validators for verifiable outcomes with LLM judges for subjective qualities. The role includes confidence-aware scoring, calibration against human-labeled trajectories, guarding against correlated errors in LLM simulators and judges, and investigating differences between simulated and production performance. The position will also build components for an improvement loop: finding failures in real use, generating realistic synthetic data and enterprise environments, attributing failures to the right part of the agent harness, proposing improvements, and validating changes against unseen cases before release. Fixed failures should become permanent tests, and the infrastructure must run reproducibly at scale while protecting organization-specific learnings and requiring admin approval for organization-specific changes. End-state outcomes are the primary guardrail.

Requirements

At least 2 years of experience in applied machine learning, data science, or ML-adjacent engineering, with a record of shipped work that was used.
Experience converting subjective human judgment into a reliable measurement that others, and ideally other models, can act on.
Strong applied machine learning fundamentals.
Comfort evaluating, prompting, and fine-tuning LLMs as components rather than pretraining them.
Strong Python skills and the ability to ship production-grade code rather than notebooks.
Ability to reason and communicate clearly about complex problems and explain what evaluation metrics mean.
High ownership and a bias toward shipping at startup pace.
Comfort with ambiguity and sound judgment about when a measurement is good enough to act on.
Experience in at least 3 of the following areas: LLM-as-judge or automated evaluation design and calibration against human judgment; human annotation programs, including rubric authoring, label quality, and annotator throughput; search ranking, recommendation systems, or online experimentation evaluation; fine-tuning and evaluating small models, including SFT, preference tuning, or distillation; reward modeling, RLHF/RLAIF, or process reward models; agent trajectory analysis and step-level fault attribution; prompt engineering as a versioned, tested, and measured engineering discipline.

Responsibilities

Design a shared evaluation judge with per-item rubrics configured alongside datasets.
Separate verifiable outcomes for deterministic validation from subjective qualities assessed by an LLM judge.
Build scoring that reports confidence and routes uncertain judgments to human review.
Maintain a calibration loop against human-labeled agent trajectories in partnership with the annotation team.
Guard against correlated blind spots between LLM-based user simulators and LLM judges.
Investigate offline-to-online divergence and keep evaluation suites refreshed with production failures.
Surface agent shortcomings in real use, including failures that appear successful.
Generate realistic synthetic scenarios and seeded enterprise environments with expected outcomes and checks.
Determine whether agent failures require changes to instructions, tools, routing, knowledge, or memory.
Optimize agent-harness components against simulation rather than production traffic.
Validate proposed changes on cases not used for tuning, detect regressions, and keep the change proposer separate from the evaluator.
Turn fixed failures into permanent tests so the evaluation suite improves with agent usage.
Build reproducible, scalable infrastructure for improvement loops across hundreds of customer organizations.
Keep organization-specific learnings appropriately scoped and ensure organization-specific changes receive admin approval.
Use end-state outcomes as the guardrail for agent improvement.