AIML - Sr Machine Learning Platform Engineer, Evaluation

Apple • Seattle, WA

2h

About The Position

Apple's AIML Evaluation team is seeking an experienced machine learning engineer to build systems that evaluate and refine Apple's foundation models and agents. The role involves designing and developing benchmarks, evaluators, simulation environments, and prompt and context optimization pipelines to improve quality across Apple's AI experiences.

The team evaluates models through offline benchmarks, device-in-the-loop simulation, and on-device production observation. It turns failures into persona-grounded datasets and reward signals for foundation-model post-training. The engineer will build systems that connect failure analysis to measurable model and agent improvements, working with product teams and the foundation-model team to identify failures, produce data, and verify results. The work supports AI products used by millions of people.

Requirements

• Strong background in machine learning and distributed systems.
• Proven ability to solve ambiguous problems end to end by defining the problem and metric, building a solution, and demonstrating measured impact.
• Experience leading technical work across multiple codebases, teams, and organizations.
• Experience building and maintaining ML infrastructure for evaluation, training, or deployment.
• At least 6 years of professional experience as a software engineer, preferably in machine learning or a related field.
• Bachelor's or Master's degree in Computer Science or a related field, or equivalent practical experience.

Nice To Haves

• Experience with LLM evaluation, LLM-as-judge, or reward modeling.
• Experience with prompt optimization, agent harness development, or post-training (SFT, DPO, RLHF).
• Proficiency in Python and ML frameworks such as PyTorch or JAX.
• Experience with agentic systems, simulation environments, or trajectory-based data generation.
• Familiarity with on-device or privacy-preserving ML.
• Understanding of how training data is used in post-training and how data choices affect model behavior.
• Excellent communication skills.

Responsibilities

• Design and develop benchmarks, evaluators, and simulation environments for Apple's foundation models and agents.
• Build prompt and context optimization pipelines that drive quality improvements across Apple's AI experiences.
• Develop systems that connect failure analysis to measurable model and agent improvements.
• Turn evaluation findings into persona-grounded datasets and reward signals for foundation-model post-training.
• Collaborate with product teams and the foundation-model team to improve model and agent quality.
• Build evaluation loops that identify failures, produce data, and check the results of improvements.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior