Software Engineer, Agent Eval Platform

ServiceNow•Mountain View, CA

4d•$113,500 - $175,900

About The Position

Moveworks is hiring a Software Engineer for its Agent Eval Platform. The role builds the measurement and evaluation infrastructure for AI agents that plan, call tools, and change state in enterprise systems. The engineer will develop evaluation rubrics and judges, calibrate scoring against human labels, and help turn evaluation signals into reward signals that can improve agent performance. This is applied machine learning and infrastructure work on open problems in evaluating language-model-based agents, rather than pretraining or conventional testing. The team is hiring across three areas: evaluation orchestration at scale, agent observability and tracing, and stateful simulation. The engineer will focus primarily on one area while contributing across the others. The work includes running multi-turn agent scenarios, capturing trajectories and outcomes, designing reliable and queryable telemetry, and building repeatable simulations of enterprise systems. The broader objective is to establish evaluation signals that can be used to optimize agents, not merely measure them.

Requirements

Experience in at least three of the following areas: distributed systems; orchestration and workflow runtimes; observability internals; concurrent and asynchronous programming; data-intensive pipelines; or gRPC/protobuf service and interface design.
0–2 years of experience building production backend or infrastructure systems.
Strong proficiency in Python or Go; ideally both.
Experience designing and operating systems that handle real traffic at scale.
Comfort making non-deterministic systems measurable and turning agent behavior into signals that engineers can use to gate releases.
Comfort working with ambiguity and novel problems without established textbook solutions.

Nice To Haves

Experience with workflow systems such as Temporal, Airflow, or Argo, or with a workflow runtime built independently.
Experience with Python asyncio, Go concurrency, or structured cancellation.
Experience with OpenTelemetry SDKs and collectors, semantic conventions, span context propagation, and high-cardinality trace data.
Experience with high-volume data ingestion, schema evolution, and sampling and retention trade-offs.
Experience designing gRPC/protobuf services and interfaces.
Interest in applying evaluation signals to agent optimization; an ML background is not required.

Responsibilities

Build the evaluation judgment layer, including rubrics, judges, calibration against human labels, and scoring methodology.
Develop runtime orchestration for end-to-end, multi-turn agent evaluation scenarios, including environment setup, simulation, transcript and trace collection, validation, scoring, and teardown.
Implement scheduling, retries, high-concurrency execution, run isolation, versioned specifications and datasets, and run-to-run comparison for evaluation workflows.
Consolidate one-off evaluation workflows into a shared orchestration service and establish reliability standards and an SLO for the harness.
Make evaluation self-serve so teams can run evaluations without bespoke integrations.
Lead the transition to OpenTelemetry-native observability for the agent platform, replacing parallel per-service logging, correlation, and redaction mechanisms.
Design the span data model for agent trajectories, including prompts, tool calls, plan updates, and outcomes, and support trace context propagation across asynchronous boundaries and long-lived sessions.
Ensure full prompts and completions are preserved through the telemetry pipeline and prevent evaluation traffic from contaminating its own data.
Build fault attribution and cross-run diffing capabilities, and develop the debugging interface and tracing contract used by support, harness, and agent teams.
Build stateful simulations of enterprise systems, backed by a datastore that persists changes within a run.
Implement per-run data injection and programmatic setup and teardown to make runs hermetic and repeatable.
Develop LLM-driven user simulators for open-ended personas and scripted state-machine simulators for deterministic flows.
Create contract-testing mocks based on real API schemas and run them in CI to detect simulation drift as vendor APIs change.
Lay the foundation for using evaluation signals to optimize agents.