ML - Data Scientist

Apple • Sunnyvale, CA

15h

About The Position

The Data Scientist will evaluate generative image and video features used on Apple devices, building the systems, human-annotation programs, and automated metrics that produce measurable and trustworthy quality signals. These signals inform model development and release decisions.

The role owns quality evaluation end to end, from evaluation-set design through annotation, scoring, analysis, and recommendations. It includes designing representative golden and evaluation sets; creating and maintaining annotation guidelines, label schemas, and lightweight review tools; monitoring annotation quality and resolving disagreements; developing and calibrating automated metrics and LLM-as-judge systems against human labels; and reporting comparisons with appropriate statistical analysis and breakdowns. Findings should identify root causes and lead to prioritized recommendations. The Data Scientist also partners with ML, engineering, and design teams to define quality standards and communicate results to technical and non-technical audiences.

Requirements

• Master's degree and a minimum of 5 years of relevant industry experience.
• At least 5 years of experience in data science, machine learning evaluation, algorithm analysis, or related technical disciplines.
• Experience designing and analyzing evaluations and experiments, especially for ML or generative-model quality, including sampling design, significance testing, and confidence intervals.
• Strong Python skills and comfort with SQL.
• Ability to build and maintain evaluation pipelines and small tools.
• Familiarity with AI-assisted analysis workflows, foundation models, agentic systems, or intelligent automation approaches for technical problem solving.

Nice To Haves

• Experience evaluating 3D Gaussian splats or diffusion models, or assessing perceptual quality.
• Experience building LLM or VLM judges and validating them against human ratings.
• Experience building web-based labeling or review tools.
• Ability to run several work streams in parallel and prioritize them.
• Proven cross-functional collaboration and ability to learn new tools quickly.
• Strong written and verbal communication, including explaining statistical tradeoffs to engineering and product partners.

Responsibilities

• Design and maintain golden sets and evaluation sets, analyzing data distributions to ensure results are representative and known defects are covered.
• Write and iterate annotation guidelines and label schemas for human evaluation.
• Build or adapt lightweight tools for image and video review.
• Monitor annotator quality and resolve disagreements and ambiguities in guidelines.
• Develop automated metrics and LLM-as-judge systems, calibrate them against human labels, and track divergences.
• Report model comparisons with confidence intervals, significance testing, and breakdowns by hotspot, content type, and failure mode.
• Translate evaluation results into root-cause findings and prioritized recommendations.
• Collaborate with ML, engineering, and design partners to define quality standards.
• Communicate results clearly to technical and non-technical audiences.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level