Director, Evaluation and Quality Assurance, Services Special Projects

Apple•Cupertino, CA

About The Position

Build and lead the evaluation and quality assurance function for a multi-year engineering initiative. This is a first-key-hire role with no existing team, tooling, or process to inherit. The Director will define how quality is measured across the product, build or adapt evaluation and automation infrastructure, and establish and lead the organization responsible for this work. The role owns evaluation and quality assurance across every product layer: determining whether the system is triggered for the appropriate user requests; assessing whether it surfaces and ranks the right results; and verifying that underlying transactions execute correctly, including integrations with external partner APIs and financial systems. Quality spans conversational understanding, relevance and ranking, and revenue-bearing correctness across a large, constantly evolving real-world dataset. The Director will create and drive the product's evaluation and quality strategy and implementation, using automated grading systems and human evaluation programs where appropriate. Responsibilities include defining correctness and quality standards, establishing continuous evaluation pipelines with clear release-linked pass/fail gates, and building the team, tooling, and operating rhythm needed to identify issues before they reach users. The role involves significant latitude and ambiguity, setting strategy, process, and technical direction while building and scaling an organization. The Director will partner closely with engineering, applied ML, and program management leadership, including senior executives, and help define what is done and shippable as the product evolves and after it launches.

Requirements

12+ years of experience in data science, machine learning evaluation, quality assurance, or test engineering roles in a technical product environment.
Experience building and leading multi-level QA or evaluation organizations through managers.
Demonstrated experience building an evaluation or QA function from the ground up, including defining metrics, process, tooling, and organizational structure.
Extensive experience with LLM-based autograders and rubrics, including rubric design, evaluation set development, validation, calibration, and human-model alignment, and using them to drive product and model improvements.
Deep experience in traditional human evaluation methodology, logging, and A/B testing for consumer-facing products at scale.
Experience evaluating conversational AI, recommendation, ranking, or search systems.
Bachelor of Science in Computer Science, Engineering, or a related technical field, or equivalent experience.
Experience with quality assurance for third-party API integrations.
Experience with logging infrastructure and instrumentation for AI product quality measurement.
Exceptional written and verbal communication skills, including presenting quality status, risks, and trade-offs clearly to senior executive leadership and influencing decisions.

Nice To Haves

Experience evaluating agentic or tool-use systems that take real-world actions, not just generate responses.
15+ years of experience in data science, machine learning evaluation, quality assurance, or test engineering roles in a technical product environment, including building and leading multi-level QA or evaluation organizations through managers.

Responsibilities

Build and lead the product evaluation and quality assurance function from the ground up.
Define quality measures and standards across the entire product, from request triggering and conversational understanding through results, ranking, transactions, partner API integrations, and financial systems.
Create and drive the evaluation and quality strategy and its implementation.
Build or adapt automation and evaluation infrastructure to operate at scale.
Determine where automated grading systems and human evaluation programs are appropriate and how to invest in each.
Define what constitutes correct and good product behavior.
Implement continuous evaluation pipelines with clear pass/fail quality gates tied to releases.
Build and scale the team, tooling, processes, and operating rhythm to identify issues before they reach users.
Set strategy, process, and technical direction in an environment with minimal existing scaffolding.
Partner with engineering, applied ML, and program management leadership, including senior executive stakeholders.
Communicate quality status, risks, and trade-offs to senior executive leadership and influence decisions.
Help define product completion and release readiness, and iterate on production decision points after launch.