Lead AI Quality Engineer

Phase2

4d•$120,000 - $145,000•Hybrid

About The Position

The Lead AI Quality Engineer owns the evolution of quality assurance at Phase2. Working closely with leaders within the Engineering Department, including other Quality Specialists, you will lead setting the vision for how Phase2 maintains a high level of quality in the software we produce, in an era when our engineers’ use of agentic AI has led to unprecedented rates of code production. You will also own setting and executing a roadmap that implements this vision: establishing consistent guidelines, methodologies, and tools that serve as the standard for quality assurance across our projects, and building a quality system capable of keeping pace with AI-accelerated delivery. Your ownership of quality assurance spans two fronts: testing our AI-powered products and features, and applying AI-driven tools and techniques to test our legacy software systems. This includes establishing standards and guidelines for creating test plans for AI-powered features (LLM-based applications, agentic workflows, ML-driven components), and for using AI-powered testing tools responsibly with appropriate human oversight. Your ownership of quality strategy runs end-to-end, from requirements and design through production monitoring, across complex digital and AI-powered delivery engagements. This is a hybrid role: allocated half-time to work on client projects, and half-time to overseeing and growing the Quality practice. Your direct work with clients involves leading quality strategy design and execution across one or more client delivery teams, presenting quality outcomes directly to client stakeholders, and coaching Phase2 project team members and client teams toward a quality ownership mindset. Your Quality practice oversight responsibilities include setting the vision and roadmap, executing on that plan, and managing, mentoring, and coaching other members of the quality team. In collaboration with the VPs of Engineering, you will help socialize our approach to quality across other departments, including project management, strategy, sales, and marketing, ensuring that everyone has a shared understanding of the vision and how it is being executed.

Requirements

8+ years of software QA experience, preferably with demonstrated experience testing AI/ML-powered features.
Experience managing, mentoring, and coaching direct report employees.
Experience recommending and then executing improvements to QA methods and tools.
Proficiency in writing automated tests in at least one framework.
Working knowledge of AI testing concepts, including hallucination detection, output evaluation, prompt engineering for test generation, and bias testing.
Strong verbal and written communication skills, with the ability to present quality outcomes both to technical and to non-technical stakeholders.
Experience with WCAG 2.1/2.2 accessibility standards.

Nice To Haves

Agency or consulting background (preferred): Experience working in a digital agency, technology consultancy, or professional services environment.

Responsibilities

Set the vision and roadmap for software quality. Define the standards for how Phase2 ensures quality in the AI era, from how we write tickets to how we test our work. Evaluate our current suite of AI-enabled and traditional testing tools, and our current QA practices and methods, and maintain a roadmap that outlines planned improvements to our tools and methods.
Define testing standards and defaults. Set the standards and defaults for what gets tested manually by humans, what is handled through automation, and what human oversight of that automation looks like. Lead this work in partnership with Engineering leaders, supported by AI Quality Specialists.
Innovate approaches to accelerate quality assurance testing. Identify and implement tools and methodologies to enable testing to keep pace with AI-accelerated software development.
Lead the use of agentic AI to write tests. Teach people across a variety of roles how to use agentic AI to write automated tests.
Recommend improvements to execution of projects and the SDLC. Identify how quality should be integrated through the project lifecycle, including when and how QA specialists should be involved at each stage of a project.
Socialize the quality assurance approach across the company. Educate and enroll other departments in the company about our evolving approach to QA, ensuring everyone has a shared vision and understanding.
Oversee agentic QA tooling. Build, run, maintain, and enhance agentic AI tools that do QA work and contribute those to Phase2’s AI delivery framework.
Test Phase2-built AI tools. Evaluate Phase2's internally built agentic AI tools (both for internal use and client use) and identify areas for improvement with clear, specific detail.
Set approaches for testing AI features. Set guidelines for designing and executing test plans for AI-powered features, including LLM-based applications, agentic workflows, and ML-driven components.
Develop AI evaluation frameworks. Develop evaluation frameworks for AI outputs, covering accuracy, relevance, consistency, hallucination detection, and bias across protected attributes.
Own integration of testing with automated development workflows. Define and help implement improvements to our CI/CD testing tools, including traditional unit, functional, integration, and VRT tests, as well as agentic AI-enabled tests.
Advise clients on quality best practices. Coach client engineering teams to adopt a quality ownership mindset, acting as a quality consultant to client stakeholders.
Lead the quality team. Manage, mentor, and coach other Quality Specialists and Quality Engineers.
Follow the cutting edge of quality engineering and share knowledge back to Phase2. Stay current on AI testing methodologies, responsible AI principles, and emerging tooling, bringing insights back to the team proactively.