Sr. Manager, Test Automation & AI Evaluation
About The Position
The Sr. Manager, Test Automation & AI Evaluation is an automation-first quality leader responsible for quality engineering across the Ignite Growth Platform. As engineering velocity has accelerated with AI-assisted development, quality assurance has become the rate-limiting constraint; this role exists to remove that constraint. The Sr. Manager makes automation the default, builds the evaluation discipline required to test AI-powered and agentic features, and applies AI to the practice of QA itself. This role also provides direct leadership of the existing Educate quality assurance team and serves as the head of quality assurance across Ignite, owning quality standards, test strategy, and release-readiness governance organization-wide.
Requirements
- 7+ years of experience in quality engineering or test automation, including team leadership or management.
- Deep, current, hands-on expertise in modern test automation frameworks such as Playwright, Cypress, or Selenium.
- Demonstrated experience building automated test suites and integrating them into CI/CD pipelines.
- Experience designing evaluation or testing approaches for AI/ML or LLM-powered systems, or a strong data-and-metrics orientation with the ability to build one.
- Strong understanding of test strategy, defect management, and release-readiness governance across multiple teams.
- Proven ability to lead, mentor, and grow a quality engineering team.
- Excellent communication skills and the ability to translate quality risk into business terms.
Nice To Haves
- Experience testing generative AI or agentic products, including evaluation harnesses, prompt testing, and regression testing for AI flows.
- Experience in a healthcare, health IT, or other regulated delivery environment where quality standards are elevated.
- Familiarity with quality management systems and traceability requirements.
- Experience with AI-assisted testing tooling and self-healing automation platforms.
Responsibilities
- Lead an automation-first quality strategy, eliminating manual testing waste and making automated testing the default across teams.
- Own modern test automation frameworks and end-to-end automated coverage, with deep, current expertise in Playwright and comparable frameworks.
- Build and own evaluation frameworks for LLM-powered features and agentic systems, covering correctness, hallucination, prompt drift, and behavioral regression.
- Establish and maintain evaluation methodology such as curated golden sets, LLM-as-judge approaches calibrated against subject-matter experts, and continuous learning loops that feed production failures back into the benchmark.
- Apply AI to QA itself, including self-healing automation, AI-assisted test generation, and intelligent defect triage.
- Own standardization of quality engineering practices across the organization, driving accountability for test coverage, defect management, release readiness, and overall delivery quality.
- Provide direct management, mentorship, and structure to the existing Ignite quality assurance function.
- Support the development of a scalable Quality Management System to ensure consistency, traceability, and process rigor across teams.
- Partner with engineering, product, and DevOps to define acceptance criteria and embed quality gates into the delivery lifecycle.
Benefits
- Health Insurance (medical, dental, and vision coverage)
- Paid Time Off (including vacation, sick leave, and flexible holiday days)
- 401(k) Retirement Plan with employer matching
- Life and Disability Insurance
- Employee Assistance Program (EAP)
- Commuter and/or Transit Benefits (if applicable)
- Discretionary company bonus
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Manager
Education Level
No Education Listed
