Machine Learning Engineer, Evaluation and Quality, Creativity Apps

Apple

1d

About The Position

The Creativity Apps team is seeking a Machine Learning Engineer focused on the quality and evaluation of machine learning models and AI features in creativity applications. The role defines how product and AI features are evaluated for release by developing evaluation strategies and metrics, building datasets and evaluation infrastructure, and driving quality from model development through customer-facing features.

The engineer owns evaluation and quality for AI features across Creativity Apps, partnering with machine learning engineering, infrastructure, product, and design teams. They translate model and use-case requirements into evaluation methods, testing strategies, and quality metrics that reflect creators’ experiences. The role also applies machine learning, LLMs, and agentic workflows to improve evaluation efficiency, accuracy, and scalability, and communicates results as release risks and recommendations.

Requirements

• At least 5 years of industry experience in machine learning or software engineering, focused on ML/AI model evaluation and quality.
• Experience with evaluation methodologies for ML models, LLMs, and AI products or features.
• Strong understanding of the ML development cycle, including data, training, evaluation, deployment, and monitoring.
• Solid machine learning fundamentals and LLM knowledge, including agentic workflows, and awareness of developments in the field.
• Track record of applying LLMs or agentic workflows to ML evaluation and quality.
• Experience in data curation, dataset creation, dataset quality, and iterative workflows for producing product-specific datasets.
• Experience designing, implementing, maintaining, or optimizing ML evaluation infrastructure at scale.
• Experience designing evaluation metrics based on the model, product, and use cases.
• Experience driving end-to-end evaluation and quality for AI products.
• Ability to report quality and present quality data to technical and non-technical audiences.
• Strong communication skills and experience working with and driving cross-functional partners.
• Proficiency in Python and/or Swift or comparable languages.
• Bachelor’s degree in Computer Science, Data Science, Machine Learning, or a related field, or equivalent experience.
• Knowledge of the software development life cycle and software quality assurance methodologies.
• Experience in objective image and video quality evaluation and metrics.

Nice To Haves

• Knowledge of color science and/or experience with subjective image or video quality review.
• Passion for creative applications; experience as a creator in photo, video, music, or design.
• Familiarity with iOS and macOS.

Responsibilities

• Define evaluation strategies and metrics for ML models, LLMs, and AI features based on the model, product, and use cases.
• Curate and create evaluation datasets, measure and improve dataset quality, and run iterative improvement loops to keep datasets representative of each product.
• Design, build, maintain, and optimize evaluation infrastructure at scale, including reproducible runs, tracked baselines, and comparisons between model versions.
• Apply ML, LLMs, and agentic workflows to evaluation, including automated judging, test generation, and analysis.
• Drive end-to-end evaluation and quality of AI features, from model checkpoints to integrated, shipping features; triage issues and drive them to resolution.
• Report quality clearly and turn evaluation results into ship/no-ship risks and recommendations for technical and non-technical audiences.
• Partner with and drive cross-functional teams, making informed judgments in ambiguous situations.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior