AI Software Evaluation Engineer, Siri AI Quality

Apple • Cupertino, CA

2h

About The Position

Join Apple’s Siri AI Quality team to help deliver high-quality AI experiences to users worldwide. The team owns the end-to-end Siri pre-ship evaluation dataset and the synthetic personas that support it. In this role, you will combine agentic engineering, data analytics, and AI evaluation to identify quality issues early, automate repetitive evaluation operations, and expand coverage as Siri’s features and platforms grow.

You will build systems that evaluate Siri’s AI experiences and translate results into action. This includes investigating test regressions and recurring issues, deriving root causes and recommendations from failure and bug data, identifying evaluation coverage gaps, and building and reporting on dataset quality, issue trends, and coverage. You will also design agentic workflows to propose, ground, validate, triage, and nominate evaluation scenarios, and extend automated screening, triage, and fix tooling to additional domains and device platforms. You will work with engineering, product, evaluator, persona, and data-platform teams to define quality standards and deliver evaluation systems that guide Siri’s development.

Requirements

• At least 5 years of industry experience in software engineering, quality engineering, data engineering, or ML engineering.
• At least 2 years of hands-on experience building LLM-based applications or agents, including tool use, multi-step workflows, and prompt engineering.
• Experience designing and running AI or LLM evaluations, including building evaluation datasets, defining metrics and rubrics, using LLM-as-judge or automated evaluators, and analyzing failures.
• Strong data analytics skills, including SQL and Python, and experience turning large datasets into clear insights and dashboards.
• Strong Python programming skills and experience with automation and CI/CD.
• Strong communication and collaboration skills, including the ability to influence cross-functional partners and work across teams.

Nice To Haves

• Experience with agent frameworks and developer tooling, such as MCP servers, skills or plugins, and agentic testing.
• Experience with voice assistants, conversational AI, or multi-turn dialogue evaluation.
• Experience with data schemas, validation frameworks, and structured data formats such as JSON.
• Experience with data visualization and BI tools such as Tableau, and data warehouses such as Snowflake.
• Familiarity with coverage analysis, traffic or taxonomy analysis, and root-cause analysis of flaky or regressing tests.
• Comfort working in a fast-paced, collaborative environment with shifting priorities.

Responsibilities

• Investigate why test scenarios regress and identify recurring issues.
• Analyze failure and bug data to develop root-cause findings and recommendations for product, engineering, and partner teams.
• Identify evaluation coverage gaps across app intents, features, and product areas, and partner with domain owners to close them.
• Build and report on dataset quality, issue trends, and evaluation coverage.
• Design agentic workflows to propose, ground, validate, triage, and nominate new evaluation scenarios with minimal manual effort.
• Extend automated screening, triage, and fix tooling to additional domains and device platforms.
• Collaborate with engineering, product, evaluator, persona, and data-platform teams to define quality standards and deliver evaluation systems.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed