AIML - Sr Machine Learning Platform Engineer, Evaluation

Apple • Seattle, WA

1d

About The Position

Join Apple’s AIML Evaluation team to build systems that evaluate and refine Apple’s foundation models and agents. The role involves designing and developing benchmarks, evaluators, simulation environments, and prompt and context optimization pipelines that improve quality across Apple’s AI experiences. You will collaborate with product and foundation model teams to connect evaluation findings with improvements, contributing datasets, environments, and reward signals that support model and agent quality.

The team evaluates models through offline benchmarks, device-in-the-loop simulation, and on-device production observation. It uses failures and traces to create persona-grounded datasets and reward signals for foundation model post-training. You will build systems that connect failure analysis to measurable improvements across Apple AI products, with a feedback loop that finds failures, produces data, and checks results. The work reaches millions of users.

Requirements

• Strong background in machine learning and distributed systems.
• A proven track record of solving ambiguous problems end to end, including defining the problem and metric, building the solution, and demonstrating measured impact.
• Experience leading technical work across multiple codebases, teams, and organizations.
• Experience building and maintaining ML infrastructure for evaluation, training, or deployment.
• At least 6 years of professional experience as a software engineer, preferably in machine learning or a related field.
• Bachelor’s or Master’s degree in Computer Science or a related field, or equivalent practical experience.
• Experience with LLM evaluation, LLM-as-judge, or reward modeling.
• Experience with prompt optimization, agent harness development, or post-training (SFT, DPO, RLHF).
• Proficiency in Python and ML frameworks such as PyTorch or JAX.
• Experience with agentic systems, simulation environments, or trajectory-based data generation.
• Familiarity with on-device or privacy-preserving ML.
• Understanding of how training data is used in post-training and how data choices affect model behavior.
• Excellent communication skills.

Responsibilities

• Design and build evaluation infrastructure for agents and foundation models.
• Set technical direction for the evaluation system and bring partner teams along.
• Develop LLM judges and reward models calibrated against human feedback.
• Develop prompt and context optimization pipelines.
• Build and integrate simulation environments for agent evaluation and trajectory-based data generation.
• Work with product and research teams to identify, prioritize, and address quality gaps.
• Build shared foundations such as run lineage, a run registry, and MCP interfaces for client teams’ agents.
• Connect failure analysis to measurable model and agent improvements.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior