AI Evaluations Engineer, US Decision Intelligence

Apple•Austin, TX

20h

About The Position

Apple’s US Decision Intelligence team is seeking an AI Evaluations Engineer to build and operate end-to-end evaluation pipelines for AI products and agentic workflows. The role owns evaluation frameworks, instrumentation, workflows, and release standards to measure AI system performance, identify failures, and track improvement over time. It supports existing AI roadmaps and develops new AI projects and experiences that improve time to insight and decision-making. The engineer will architect an evaluation framework that traces performance across agent responses, skills, and tool calls, with a focus on accuracy, performance, latency, and A/B testing. The role defines platform-wide evaluation gates and pass/fail standards, including authority to hold a release when standards are not met. It works with AI engineers, platform teams, product and domain experts, track leads, and the Evals team in India to establish measurable evaluation criteria and consistent implementation across geographies. The role also contributes to observability, retries, and logging system design.

Requirements

At least 5 years of experience in data- and AI-related fields such as AI engineering, software development, ML engineering, data science, or QA.
Strong Python skills.
Hands-on experience with AI evaluation techniques such as golden datasets, LLM-as-a-judge, or rubric-based scoring.
Experience with LLM ecosystems such as OpenAI, Anthropic, or Gemini; RAG pipelines; and vector databases such as Pinecone, FAISS, Milvus, or PostgreSQL.
Proficiency in SQL and experience with at least one major data analytics platform, such as Hadoop, Spark, or Snowflake.
Experience with CI/CD or release validation workflows.
Experience working with data science teams on insights generation using LLMs.
Hands-on experience with Langfuse or similar LLM observability tools.
Comfort translating fuzzy correctness criteria with product or domain experts into measurable rubrics or metrics.
B.S. degree in Computer Science or Engineering, or equivalent work experience.
Sound communication skills, including the ability to present domain and technical content appropriately for the audience.
Ability to gain the trust of stakeholders and senior leadership.
Familiarity with embeddings, retrieval algorithms, agents, and data modeling for vector and graph databases.
Ability to manage time, collaborate across multiple teams, balance competing priorities and long-term projects with ad hoc requirements, and work in a fast-paced, dynamic, evolving business environment.
Eagerness and ability to learn new skills and solve dynamic problems.

Nice To Haves

Experience with distributed systems architecture and asynchronous messaging, agent communication, and caching technologies such as RabbitMQ, Redis, and Valkey.
Experience working across global teams to ensure product-development alignment.
Applied knowledge of GenAI and RAG strategies, microservices, recommendation systems, and context engineering.
Working knowledge of agent evaluation concepts, including trajectory, end-to-end, and component-level evaluation, and tool-call correctness.
Advanced degree (MS or Ph.D.) in Economics, Electrical Engineering, Statistics, Data Science, or a similar quantitative field.

Responsibilities

Build and operate AI evaluation workflows measuring LLM output quality across chat, summarization, recommendations, and agentic actions.
Architect a comprehensive evaluation framework tracing agent responses, skills, and tool calls, and use it to optimize accuracy, performance, and latency.
Run A/B tests to identify the best recommendations across the system.
Implement rubric-based evaluations for correctness, relevance, grounding, and consistency.
Develop agent-as-a-judge and harness-as-a-judge patterns, including evaluation of real traces in a sandbox.
Instrument LLM and agent workflows to capture traces, prompts, responses, metadata, and user feedback.
Own the platform-wide evaluation gate, define release pass/fail contracts, and hold releases that do not meet the standards.
Define agent-specific evaluations, including task completion, tool correctness, and error recovery.
Partner with AI engineers and AI platform teams to turn product requirements into evaluation criteria.
Arbitrate evaluation disputes with track leads and establish standards for geo evaluation engineers to implement.
Define rerun policies and variance thresholds.
Collaborate with the Evals team in India to maximize global impact.
Contribute to system design for observability, retries, and logging.