Software Engineer, Agent Eval Platform

ServiceNow•Mountain View, CA

4d•$113,500 - $175,900•Onsite

About The Position

Moveworks is hiring a Software Engineer to build the judgment layer of its agent evaluation platform. The role focuses on measuring multi-step AI agent behavior in stateful enterprise systems, including developing evaluation rubrics and judges, calibrating scores against human labels, and creating methods that make evaluation results meaningful enough to improve and train agents. The work is applied engineering and machine learning methodology, rather than pretraining or conventional testing, in an area where evaluation methods are still evolving. The position spans three areas: evaluation orchestration at scale, agent observability and tracing, and stateful simulation. The engineer will anchor in one area while contributing to others. Across the areas, the work aims to establish reliable evaluation infrastructure and lay the foundation for using evaluation signals to optimize agents. The role is listed as required in office. Compensation is a base pay of $113,500–$175,900, plus equity when applicable, variable or incentive compensation, and benefits. Individual compensation varies based on qualifications, skills, competencies, and work location.

Requirements

0–2 years of experience building production backend or infrastructure systems.
Strong proficiency in Python or Go; ideally both.
Experience designing and operating systems that handle real traffic at scale.
Experience in at least three of the following areas: distributed systems; orchestration and workflow runtimes; observability internals; concurrent and asynchronous programming; data-intensive pipelines; gRPC/protobuf service and interface design.
Comfort making non-deterministic systems measurable and turning fuzzy agent behavior into signals engineers can use to gate releases.
Comfort working with ambiguity on novel problems without established textbook solutions.

Responsibilities

Build and maintain the judgment layer for the agent evaluation platform, including evaluation rubrics, judges, calibration against human labels, and scoring methodology.
Develop runtime infrastructure to execute multi-turn agent scenarios end to end, including environment and user-simulator setup, interaction loops, collection of transcripts, traces, and final state, validation, scoring, and teardown.
Implement scheduling, retries, high-concurrency execution, run isolation, versioned specifications and datasets, and run-to-run comparisons for evaluation workflows.
Consolidate existing one-off evaluation workflows into a single orchestration service and establish reliability goals and self-service evaluation capabilities.
Lead adoption of OpenTelemetry-native observability for the agent platform, replacing parallel logging, correlation, and redaction mechanisms.
Design the span data model for agent trajectories, including prompts, tool calls, plan updates, and outcomes; support trace-context propagation across asynchronous boundaries and long-lived sessions.
Ensure prompts and completions remain intact through the pipeline and prevent evaluation traffic from contaminating its own data.
Build fault-attribution, cross-run comparison, and debugging capabilities for support and harness engineers, and define tracing interfaces with the agent-building team.
Build stateful simulations of enterprise systems such as ITSM, HR, knowledge bases, and inventory, backed by a datastore that persists changes within a run.
Implement per-run data injection and programmatic setup and teardown to make evaluations hermetic and repeatable.
Develop LLM-driven user simulators for open-ended personas and scripted state-machine simulators for deterministic flows.
Create contract-testing mocks against real API schemas in CI to detect simulation drift as vendor APIs change.
Develop isolated sandbox environments that reproduce agent configuration, identity, search content, and permissions from a consistent baseline, with teardown after each run.
Lay the foundation for using evaluation signals to optimize agents, not only measure them.