Senior Manager, Evaluation, Services Special Projects

Apple•Cupertino, CA

11h

About The Position

Apple is seeking a hands-on Evaluation leader to build and lead a team of applied scientists and machine learning engineers measuring the quality of a new, multi-year initiative. Joining at the start, this leader will contribute to establishing how the team determines whether the system is correct, relevant, and trustworthy before and after launch. The role owns components of the evaluation plan, including methods, datasets, and automated grading systems that translate product behavior into clear quality signals for engineering and executive leadership. The leader will oversee evaluation for a conversational experience that interprets user requests, finds and ranks relevant results, composes responses, and enables task completion. They will set technical direction for LLM-based autograders and rubrics, including rubric design, evaluation-set development, validation, and calibration. Evaluation methods will be paired with logging, instrumentation, and online experimentation, in collaboration with other evaluation leaders. The role develops a roadmap for its evaluation components, translates the broader evaluation strategy into methods and deliverables, and ensures findings drive measurable quality improvements and inform release decisions. It is hands-on: the manager reviews designs and experiments, stays close to data, mentors scientists and engineers, and recruits and grows the team. The role partners with engineering, applied machine learning, quality assurance, and program management teams, and presents quality status, risks, and trade-offs to senior leadership. With the product, architecture, and quality bar still evolving, the leader will help define what constitutes good and shippable quality.

Requirements

6+ years of experience in machine learning, data science, or machine learning evaluation.
3+ years of direct people management of scientists or engineers.
Track record of hiring, developing, and leading teams of applied scientists, data scientists, or machine learning engineers.
Hands-on experience building and deploying LLM-based autograders and rubrics, including rubric design, evaluation-set development, validation, and calibration.
Experience evaluating conversational AI, search, recommendation, or ranking systems using offline metrics and online experimentation.
Strong command of statistics and experimental design, including A/B testing and interpretation of noisy metrics.
Exceptional written and verbal communication skills, with a track record of influencing engineers and senior leaders.
Proven track record of building automated grading systems that scale to large, frequently changing datasets and using them to improve models and product features.
Experience defining metrics and evaluation sets that reflect real user behavior and predict real-world quality.
Experience partnering with engineering and applied machine learning teams to diagnose failures and ship fixes based on evaluation findings.
Experience contributing evaluation results to release processes as explicit quality gates.
Experience growing senior individual contributors and technical leads and recruiting strong talent in competitive hiring markets.

Nice To Haves

MS or PhD in Computer Science, Machine Learning, Statistics, or a related quantitative field, or equivalent experience.
Experience evaluating agentic or tool-use systems that take real-world actions, including multi-turn behavior, planning, and task completion.
Experience with relevance and ranking evaluation, including graded relevance judgments and ranking metrics.
Experience with groundedness and factuality evaluation of generated responses.
Experience with logging infrastructure and instrumentation for AI product quality measurement.
Experience generating and validating synthetic evaluation data.
Experience evaluating systems that integrate with third-party partner APIs.
Experience with privacy-preserving evaluation practices.

Responsibilities

Build and lead a team of applied scientists and machine learning engineers focused on product evaluation.
Own evaluation methods, datasets, and automated grading systems for assigned product components.
Set technical direction for LLM-based autograders and rubrics, including rubric design, evaluation-set development, validation, and calibration.
Design and run evaluation for a conversational experience, covering request interpretation, retrieval and ranking, response quality, and task execution through external integrations.
Build a roadmap for owned evaluation components and translate the broader evaluation strategy into concrete methods and deliverables.
Pair evaluation methods with logging, instrumentation, and online experimentation in collaboration with other evaluation leaders.
Turn evaluation findings into measurable quality gains and clear inputs to release decisions, including explicit quality gates.
Review designs and experiments and remain close to the data.
Mentor scientists and engineers, recruit talent, and grow senior individual contributors and technical leads.
Partner with engineering, applied machine learning, quality assurance, and program management teams to diagnose failures and ship fixes.
Present quality status, risks, and trade-offs to senior leadership.
Help shape the evolving product quality bar and define what constitutes good and shippable quality.