AI Software Evaluation Engineer, Siri AI Quality

Apple • Cupertino, CA

1d

About The Position

The AI Software Evaluation Engineer will help Apple’s Siri AI Quality team deliver high-quality AI experiences. The team owns the end-to-end Siri pre-ship evaluation dataset and the synthetic personas that power it. This role combines agentic engineering, data analytics, and AI evaluation to identify quality issues early, automate repetitive evaluation operations, and expand coverage as Siri features and platforms grow.

The engineer will build systems that evaluate Siri AI experiences and translate results into actionable improvements. The work includes investigating regressions and recurring issues, analyzing failures and bug data to identify root causes, assessing coverage gaps across app intents, features, and product areas, and developing reports on dataset quality, issue trends, and coverage. The role also involves designing agentic workflows to propose, ground, validate, triage, and nominate evaluation scenarios, as well as extending automated screening, triage, and fix tooling to additional domains and device platforms.

The engineer will collaborate with engineering, product, evaluator, persona, and data-platform teams to define quality standards and deliver evaluation systems that guide Siri’s development.

Requirements

• At least 5 years of industry experience in software engineering, quality engineering, data engineering, or ML engineering.
• At least 2 years of hands-on experience building LLM-based applications or agents, including tool use, multi-step workflows, and prompt engineering.
• Experience designing and running AI or LLM evaluations, including building evaluation datasets, defining metrics and rubrics, using LLM-as-judge or automated evaluators, and analyzing failures.
• Strong data analytics skills, including SQL and Python, and experience turning large datasets into clear insights and dashboards.
• Strong Python programming skills and experience with automation and CI/CD.
• Strong communication and collaboration skills, including the ability to influence cross-functional partners and work across teams.
• Comfort working in a fast-paced, collaborative environment with shifting priorities.

Nice To Haves

• Experience with agent frameworks and developer tooling, such as MCP servers, skills or plugins, and agentic testing.
• Experience with voice assistants, conversational AI, or multi-turn dialogue evaluation.
• Experience with data schemas, validation frameworks, and structured data formats such as JSON.
• Experience with data visualization and BI tools such as Tableau and data warehouses such as Snowflake.
• Familiarity with coverage analysis, traffic or taxonomy analysis, and root-cause analysis of flaky or regressing tests.

Responsibilities

• Investigate why test scenarios regress and identify recurring issues.
• Analyze failure and bug data to develop root-cause findings and recommendations for product, engineering, and partner teams.
• Identify evaluation coverage gaps across app intents, features, and product areas, and partner with domain owners to address them.
• Build and report on dataset quality, issue trends, and evaluation coverage.
• Design agentic workflows to propose, ground, validate, triage, and nominate evaluation scenarios with minimal manual effort.
• Extend automated screening, triage, and fix tooling to new domains and device platforms.
• Collaborate with engineering, product, evaluator, persona, and data-platform teams to define quality standards and deliver evaluation systems.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed