Systems Engineer - Evaluation Engineering

Apple • Cupertino, CA

19h

About The Position

The Systems Engineer will own infrastructure for the Siri Agentic Evaluation Platform, which supports high-throughput agentic simulations, multi-model judging pipelines, and real-time observability across large volumes of tokens and complex data types. The role involves designing and building an execution engine that coordinates evaluation loops while maintaining determinism, fault tolerance, and cost efficiency across parallel requests and heterogeneous Apple devices, including iPhone, Mac, and iPad.

The engineer will build internal self-service infrastructure for ML and product teams, develop performant APIs and trace-data services, and create data pipelines that preserve immutable lineage from evaluation metrics to raw generations. The role also includes managing deployments across multi-tenant clusters and implementing observability, scheduling, and protective controls to mitigate rate limits and cascading failures.

Requirements

• Master's degree in computer science or equivalent.
• At least 7 years of experience as a distributed systems engineer, platform engineer, or equivalent.
• Strong proficiency in concurrent, enterprise-scale languages such as Python with asyncio or Java.
• Deep expertise designing robust, versioned production APIs using gRPC/Protobuf, GraphQL, or REST (FastAPI).
• Advanced experience with Kubernetes, including orchestration, custom operators, and service meshes such as Istio or Linkerd, and with a cloud provider such as AWS, GCP, or Azure.
• Strong experience modeling complex relational data and trace hierarchies using PostgreSQL, along with high-throughput analytical query layers.
• Experience designing asynchronous, event-driven architectures using Kafka, AWS SQS/SNS, RabbitMQ, or Redis Streams.
• Proficiency with Terraform for declarative infrastructure management.
• Experience building automated, containerized deployment pipelines using GitHub or ArgoCD, with an emphasis on fast and reliable developer feedback loops.

Nice To Haves

• Experience building Agentic RAG platforms or developer-facing infrastructure tooling.

Responsibilities

• Architect and scale the asynchronous execution engine for parallel agent simulations, validation tests, and LLM-as-a-judge pipelines.
• Design and build self-service infrastructure, CLI tools, and internal APIs that help ML and product teams integrate evaluation pipelines into CI/CD workflows.
• Design, build, and maintain performant, type-safe APIs for evaluation pipelines, trace data, and real-time generation metrics.
• Build data pipelines to ingest and transform high-volume execution traces, preserving immutable lineage from metrics to raw generations.
• Own deployment topologies across multi-tenant clusters using declarative infrastructure and continuous delivery practices.
• Implement distributed tracing, structured metrics, and alerting across the platform.
• Design scheduling layers, token buckets, and circuit breakers to prevent downstream API rate limiting and cascading cluster failures.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior