AI/LLM Quality Engineer - Siri

Apple

8d

About The Position

The AI/LLM Quality Engineer will elevate the quality of Siri features and intelligent services in Mainland China. The role owns product quality across the development lifecycle, combining traditional software quality assurance with Generative AI and LLM evaluation methods, while building automation tools to improve testing and engineering efficiency.

Working at the intersection of AI, speech and language technologies, and customer experience, the engineer will develop test strategies and evaluation systems; assess conversational AI, retrieval-augmented generation (RAG), tool-use/agent, and multimodal scenarios; investigate failures across client, backend, API, telemetry, and model layers; and collaborate with engineering, product, localization, and cross-site teams. The role also supports content safety, privacy, bias, and responsible AI evaluations.

Requirements

• Bachelor’s or master’s degree in Computer Science, Computer Engineering, or equivalent practical experience.
• Strong proficiency in at least one modern programming or scripting language, such as Python, Swift, Java, or Go.
• Solid fundamentals in data structures, databases, SQL, and Unix, Linux, or macOS environments.
• Hands-on experience architecting and implementing test automation frameworks for client-side applications, backend services, and end-to-end integration flows.
• Experience with CI/CD pipelines, containerization, and modern cloud or distributed infrastructure.
• Solid understanding of Generative AI and LLM concepts, with demonstrated ability to build tools, scripts, or workflows using model APIs.
• Strong analytical, logical, debugging, and root-cause analysis skills, including the ability to investigate issues across diverse technical stacks.
• Ability to adapt and remain resilient amid shifting priorities in a fast-paced environment.
• Strong communication and interpersonal skills, including effective collaboration across cross-functional and geographically distributed teams.
• Professional working proficiency in written and spoken English and Mandarin.
• Demonstrated ability to establish LLM-driven lifecycle QA systems and automated evaluation pipelines.
• Ability to define model quality metrics and dashboards, analyze failure cases, and work with cross-functional teams to resolve them.
• Experience designing evaluations and benchmarks for conversational AI, RAG, tool-use/agents, and multimodal scenarios.
• Experience evaluating content safety, compliance, privacy, and responsible AI risks, including bias and harmful content generation.
• Self-motivated and proactive, with creative and critical thinking capabilities.

Nice To Haves

• Hands-on experience developing lightweight AI-driven developer productivity tools, such as agentic workflows or LLM-assisted code or test generation.
• Experience testing on Apple platforms, including iOS, iPadOS, macOS, and watchOS.
• Fluency or proficiency in Cantonese.

Responsibilities

• Drive test strategies, comprehensive test plans, design reviews, and quality execution for new Siri capabilities and services.
• Design evaluation schemes, curated benchmark datasets, adversarial and negative samples, and edge cases across conversational AI, RAG, tool-use/agents, and multimodal scenarios.
• Develop internal AI-enabled tools, test-generation pipelines, and self-healing test frameworks to improve operational velocity and engineering productivity.
• Define model quality metrics and monitoring dashboards; collect, categorize, and diagnose failure modes, and partner with modeling and product teams to drive improvements.
• Investigate and troubleshoot complex issues across client applications, cloud backend services, APIs, telemetry and logs, and AI model behaviors.
• Support evaluations for content safety, privacy, bias, and responsible AI risks.
• Collaborate with engineering, product, localization, and cross-site teams to promote quality practices throughout the software development lifecycle.
• Establish LLM-driven lifecycle QA and automated evaluation pipelines, including test-case generation, offline model evaluation, self-healing end-to-end automation, online monitoring, and A/B experimentation quality assessment.
• Assess Apple-platform software and services.

What This Job Offers

• Job Type: Full-time