Senior Manager of Meta Evaluation & Quality Assurance

Apple • Seattle, WA

3d

About The Position

Apple Services Engineering (ASE) powers many AI and LLM features across App
Store, Music, Video and more. As these systems increasingly rely on
human-in-the-loop evaluation systems, the quality of our decisions is
constrained by the quality of our evaluation systems. We believe that to build
exceptional LLMs, you need exceptional mechanisms to validate the signals used
to train and evaluate them.

As the Senior Manager of Meta Evaluation & Quality Assurance, you will lead a
specialized team of Data Scientists and Machine Learning Engineers who evaluate
the evaluators. You will move beyond basic validation to lead the strategy and
technical development of ML-based validation frameworks and automated data
quality validation pipelines. You will set strategy, guide execution, and work
cross functionally to deliver a cohesive quality system that combines machine
learning with human-in-the-loop processes to ensure our metrics are trustworthy,
robust, and decision-ready.

Requirements

• 8+ years of experience in Data Science, Machine Learning, or Evaluation Science, with 3+ years leading technical teams
• Strong background in Meta Evaluation, AI/ML measurement, statistics, or quality assurance methodologies.
• Demonstrated success in designing Human-in-the-Loop (HITL) machine learning systems or active learning pipelines.
• Masters degree in Statistics, Data Science, Machine Learning or related field.

Nice To Haves

• PhD in Statistics, Computer Science, Machine Learning, or related field
• Deep understanding of evaluation pipelines, calibration techniques, and statistical process control
• Experience building ML models specifically designed for quality estimation, anomaly detection, or disagreement modeling
• Proficiency in Python or R for statistical analysis and reasoning about evaluation data
• Experience defining governance gates or certification processes for AI systems
• Proven ability to manage complex methodological and technical programs in dynamic, fast-paced environments.
• Exceptional communication, organizational, and analytical skill.

What This Job Offers

• Job Type: Full-time
• Career Level: Manager
• Number of Employees: 5,001-10,000 employees