AI Automation Quality Engineer

Harvard University • Cambridge, MA

4d • Remote

About The Position

The AI Automation Quality Engineer leads the next generation of quality engineering for the my.harvard portal, moving beyond traditional QA to design intelligent, agent-based testing capabilities that automatically analyze requirements, understand application code, generate reusable test assets, and continuously improve HUIT’s testing strategy. This role combines strong software testing experience with modern AI engineering concepts, including Retrieval-Augmented Generation (RAG), large language models (LLMs), agentic workflows, and test automation frameworks. Working closely with Business Analysts, developers, and product owners, the AI Automation Quality Engineer helps build an autonomous QA platform that accelerates delivery while improving software quality.

Requirements

• Minimum of two years’ post-secondary education or relevant work experience
• 3+ years building automated test frameworks for enterprise web applications
• Has used AI coding assistants in a real testing workflow — generating, maintaining, or reviewing tests — and can describe concretely where they worked well and where they failed or produced output that needed correction
• Test automation and quality engineering best practices
• Strong Python development experience
• Experience with Playwright, Selenium, Cypress, pytest, or similar frameworks
• Experience using Jira and Agile development processes
• Understanding of Git workflows and CI/CD pipelines
• Strong analytical thinking and ability to translate business requirements into automated quality strategies
• Knowledge of SDLC and STLC

Nice To Haves

• Experience developing AI agents that generate code or software tests
• Continuous improvement and adaptability to emerging AI and testing technologies
• Experience testing Django and/or FastAPI applications
• Experience in higher education or enterprise ERP/student information systems
• Experience measuring test coverage
• Cross-functional collaboration with Business Analysts, developers, and product owners
• Demonstrated team performance skills, service mindset approach, and the ability to act as a trusted advisor
• Completion of Harvard IT Academy specified foundational courses (or external equivalent) preferred

Responsibilities

• Design and implement AI agents that read Jira user stories, acceptance criteria, architectural documentation, and business rules to automatically generate comprehensive functional, integration, regression, accessibility, and API test scenarios
• Build agents capable of analyzing Django, FastAPI, and frontend codebases to understand application behavior, identify impacted components, and generate reusable test cases with high coverage
• Develop Retrieval-Augmented Generation (RAG) pipelines that leverage Jira, Confluence, Git repositories, existing test suites, architecture documentation, API specifications, and historical defects to improve test quality and reduce hallucinations
• Implement workflows where generated test scenarios are reviewed and validated by Business Analysts and Product Owners before automation code is produced
• After approval, generate maintainable automated tests using Playwright, Selenium, pytest, or equivalent frameworks, validate results, and prepare commits or pull requests for inclusion in the shared test repository
• Capture review feedback, production defects, and failed test patterns to improve prompts, retrieval strategies, reusable testing patterns, and agent performance over time
• Maintain regression suites, API tests, UI automation, performance validation, and CI/CD integration while ensuring generated tests remain reliable and maintainable
• Partner with developers, architects, BAs, and DevOps engineers to define AI-driven quality engineering standards and governance

Benefits

• Generous paid time off including parental leave
• Medical, dental, and vision health insurance coverage starting on day one
• Retirement plans with university contributions
• Wellbeing and mental health resources
• Support for families and caregivers
• Professional development opportunities including tuition assistance and reimbursement
• Commuter benefits, discounts and campus perks

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: Associate degree