Software Engineer, Evaluation

Apple • Cupertino, CA

2d

About The Position

As a Software Engineer in AIML Evaluation at Apple, you will help improve Siri and Apple Intelligence by building systems and methods to evaluate AI/ML models and product experiences. You will collaborate with product managers, backend engineers, data scientists, and other stakeholders to improve the quality and user experience of Apple products. The role includes developing evaluation infrastructure and frameworks, improving operational controls and auditability, and scaling evaluation workflows.

You will design benchmarks, metrics, and test suites for model capabilities including reasoning, knowledge, code, and agentic workflows. You will develop product-aligned evaluation methods, research evaluation techniques such as model-based judging and contamination-resistant benchmarks, and build reusable tools and analysis frameworks. You will also run experiments, analyze capability gaps, work with third-party benchmarking vendors, and collaborate with model training, training data, and product teams so evaluation findings inform development priorities and product quality.

Requirements

• At least 3 years of professional software engineering experience building and operating production systems.
• Ability to tackle complex challenges, think critically, and deliver innovative solutions.
• Fluent use of AI-assisted development tools, such as coding agents and code review assistants.
• Experience designing and scaling distributed systems, including load balancing, active-active topologies, capacity planning, and throughput-bound services.
• Experience with frontend web development using React and modern JavaScript frameworks.
• Experience with RESTful API design and SQL and NoSQL databases.
• BS or MS in Computer Science, Computer Engineering, or an equivalent degree with applicable experience.
• Ability to operate independently on ambiguous, open-ended problems.
• Experience working on large, cross-functional, multi-team projects.

Nice To Haves

• Eagerness to explore and apply cutting-edge technology solutions.
• Commitment to elevating every aspect of the user experience.

Responsibilities

• Work with product managers, backend engineers, data scientists, and other stakeholders to improve product quality and user experience.
• Design and develop evaluation systems and frameworks for Apple Intelligence models.
• Tune evaluation infrastructure, strengthen operational controls, improve auditability and debug trails, and scale workflows.
• Design and implement benchmarks, metrics, and test suites for reasoning, knowledge, code, and agentic workflows.
• Develop evaluation methods that reflect model behavior in product settings and validate whether metrics predict user-perceived quality and product outcomes.
• Research and apply evaluation techniques, including scoring frameworks, model-based judging, and contamination-resistant benchmark design.
• Build reusable tools, scorer libraries, and analysis frameworks for the team’s benchmark portfolio.
• Design and execute experiments comparing model capabilities, engage third-party vendors on benchmarking, and perform gap analyses.
• Collaborate with model training, training data, and product teams to apply evaluation insights to training strategies, data decisions, and product quality improvements.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level